Last year I edited biology, other science, and myths/legends for 2020 WORKSHOP, a set intended to be an opportunity for newer writers to write questions with the aid of mentorship as a spiritual successor to the PADAWAN set. This year, I am reprising my editing roles for 2021 WORKSHOP, as well as serving as co-head editor alongside Olivia Murton and Wonyoung Jang. I am writing this retrospective in an effort to expand the benefits of this project to the wider community and be transparent about the success and failures of our work.
Throughout this post I will frequently use “the royal we” so I can pretend that I am speaking for anyone other than myself - others involved in the process are encouraged to add their voices into the mix.
What didn’t work
I’m opening with this section to immediately address some of the pitfalls we ourselves got stuck in. Some of these were unique to our situation - some weren’t.
We did not do as good of a job of being mentors as we could have
I will take a moment to split the hair between “mentor” and “editor” for a project in which everyone was both - as editors our job was to take in raw questions and produce a final set, and as evinced by the fact that many people played 2020 WORKSHOP we succeeded in that goal. As mentors our job was to provide the best space we could for our mentees to try writing for the first time and we did not do as well on that front.
In some sense, this was unavoidable. Everyone is human, and it’s hard to sustain the same level of engagement with a project for its entirety, especially as mentorship is very taxing. I am not criticizing any of my fellow editors on either 2020 or 2021 WORKSHOP, as they all put in hard work and got good results, but there is no point in sugarcoating the fact that as a whole we could have done better.
The biggest problem was the speed of feedback. As part of the production pipeline (referred to internally as the “WORKSHOP Workflow”), writers needed to receive approval for answerlines, and would typically receive feedback that would allow them to self-edit questions before a final pass by the editor (I am aware that there are other projects that skip this first step and allow writers to produce full tossups before requiring editor oversight, but I think that both approaches have their upsides). When editors took time (days, weeks, sometimes months) to respond to proposed answerlines, or further drafts, that really hampered the ability of writers to participate in the process and stripped them of some of their agency as writers.
There is no real solution to this, but communication with editors is the biggest tool to avoid this and establish accountability. For 2021 WORKSHOP, we have stressed to our writers that they can talk to editors and even ask them to edit their questions - this should be the norm for all sets. Additionally, I did not consider myself in a position to ask my fellow editors to do things last year, and I think this is not uncommon - it is very difficult to tell people you consider your peers or even your betters that they haven’t done their job. Head editors have to take the initiative on this sort of thing.
Our organizational system was not good
2020 WORKSHOP had a very complicated system for keeping track of how much people had written and incentivizing writers to both write more and write consistently - I am very much in the debt of Will Grossman for his work in making the cogs turn. It involved tracking how much each writer had written each week and adjusting the amount they were “required” (read: asked politely) to write for the next week on the fly, with monetary bonuses given to writers who were able to meet these goals every week. It was not good - it had too many moving parts and the only value was the ability to award consistency bonuses, which unduly punishes writers who write less or write slower in favor of writers who already are getting compensated more by virtue of writing more.
For 2021 WORKSHOP, we have moved towards a more straight-forward “chunk”-based system, in which each writer chooses 4/4 to write and they can write additional chunks as they finish the previous one, and have entirely removed consistency bonuses - I believe we took inspiration from Harvard’s system when writing HFT, and I don’t doubt that other sets have done this as well. This significantly reduced day-to-day overhead and I would recommend it for this reason, as well as some I will discuss in the next section.
Some of our writers did not write and we were not proactive about our solutions
Writers saying they can write some number of questions and ultimately not doing so is a very common scenario, especially for a project like WORKSHOP where many writers are taking a step out into the unknown and committing to something they may not have done extensively before. In general though, editors and writers have no reason to worry about individuals not producing as many questions as they claimed they would, so long as the set is finished and no one is unduly inconvenienced - others will simply pick up the slack.
The problem was that we were split between being editors working towards completing a set and mentors who were trying to ensure the maximum opportunity possible for all of our writers. As part of this second goal, we had allowed each writer to explicitly state how many questions they wanted to write in their application and had split up the distribution accordingly. When a writer did not meet deadlines, we could not impel them to write in the way that one could force a more experienced writer or a teammate to do so. We could not take the other approach, either, and open their section of the distro up to other writers, because we could not meaningfully decide which questions they still had the potential to write and removing their entire claim would prevent them from writing at all. So we sat on our hands, and had to deal with the consequences as the first mirror approached.
The first solution to this was yet again communication. In this iteration of WORKSHOP we have tried much harder to be in contact with individual writers and this means that we have not needed to guess at what their intentions were - they would simply tell us. The second was to switch to the chunk system mentioned earlier, which meant that no writer could overcommit at the beginning and it was much easier to dynamically change the expectations as the writing process went on.
We overshot difficulty
Ignoring for a moment the fact that the set was accidentally announced to be half a step lower in difficulty than intended, 2020 WORKSHOP was harder than the projected difficulty. Power counts across mirrors were consistently depressed, and had we had Advanced Stats or other forms of buzzpoint analysis we would have almost certainly seen many clues unbuzzed on.
Not much we could do here but adjust expectations appropriately - both ours and the audience’s. We are more consciously writing questions to be easier for 2021 WORKSHOP, but people reading this post shouldn’t be too surprised in a year’s time when we’ve accidentally overshot our target again - everyone does it.
What did work
We were able to provide mentorship for many writers
I am very happy to see that many of the writers who worked on 2020 WORKSHOP have continued on to other projects, and I’m keeping an eye out for those that I haven’t seen yet. Despite the aforementioned inefficiencies in our process, the vast majority of the questions were written by our writer core, which is quite a feat when you consider their overall inexperience and the difficulty level of the set - it hasn’t happened since PADAWAN! 2021 WORKSHOP has fixed many of the problems of the previous year and ideally will have success as well, and I can only hope that WORKSHOP becomes a long-standing institution in the quiz bowl community.
There’s been a lot of discussion in the community recently about the role of mentorship programs, especially when compared to packet sub. I don’t think there can be any doubt that dedicated projects like WORKSHOP serve an important role, but I will briefly reiterate my stance here that both those and other avenues are necessary to reach different kinds of people.
Having non-editor people on the team is amazing
I think, sitting in the editor’s chair and looking back, the best part of 2020 WORKSHOP was having dedicated people for tasks other than editing - proofreading, logistics, mirror organization, etc. For 2021 WORKSHOP we expanded the program to include mentoring opportunities for people who want to do logistics work in quiz bowl, which means that we have an entire team of people for getting mirrors organized. Anyone who has tried to juggle editing duties with proofreading or weighing prospective hosts will know how hard it can be, so all I can say is: get yourself a logistics team, even if it’s just one person.
The spreadsheets we used for answerlines were very powerful
Any project written by more than a few people needs to have a plan for how to track things like the subdistribution, question answerlines, what writers have written, what people are owed, how much the set is written, etc. This was especially true for WORKSHOP, which had six editors and 42 writers. We were able to have all of this information integrated into a single spreadsheet, which would automatically keep track of who had written what by annotating answers in the distribution with writer and editor tags. The incredible efficiency of this document allowed editors and writers alike to just glance at a single place to figure out what tasks they had left to do or see the overall progress of the set.
If anyone would be interested in using this sheet for their projects, please contact me or one of the other head editors of WORKSHOP and we will gladly work to adapt the system to your needs.
Our points-based compensation system was very streamlined
Rather than assigning every question a dollar amount at the get-go, we made use of a point system that allowed us to divy up set profits between writers and editors after all mirrors were finished. Each tossup was worth 5 points in total and each bonus 4 - when a question was edited, three points would go to the editor and the remainder to the writer, meaning 2 for tossups and 1 for bonuses. A simplified version of this was also used for proof-reading work, though that was primarily a count of questions checked.
One feature that we decided to remove between 2020 and 2021 was the “minimal editing bonus”, which was a flag that indicated that a question had meant some standard of quality and thus one additional point would be given to the writer and one fewer to the editor. There was no way to ensure consistency in how this bonus was awarded between editors and in general it felt too subjective a measure for us to feel comfortable using it to determine payment.
The set was good
Difficulty concerns aside, I think 2020 WORKSHOP was a very good set; I was personally pleased to see specific praise of the science as someone who was responsible for roughly half of it. It is unfortunate that so few teams got to play it because of the abrupt shortening of the 2019-2020 season, but it is my hope that teams will be able to appreciate the ideas and the clues as they study the set in years to come.
2020 WORKSHOP: A Retrospective
- Santa Claus
- Rikku
- Posts: 285
- Joined: Fri Aug 23, 2013 10:58 pm
2020 WORKSHOP: A Retrospective
Kevin Wang
Arcadia High School 2015
Amherst College 2019
2018 PACE NSC Champion
2019 PACE NSC Champion
Arcadia High School 2015
Amherst College 2019
2018 PACE NSC Champion
2019 PACE NSC Champion
Re: 2020 WORKSHOP: A Retrospective
Kevin said in the main Discord that 2020 WORKSHOP didn't have a writing Discord server, which I think is something every set [or at least every set with new writers] really needs. CALISTO and Saturnalia facillitated inter-writer/editor communication by keeping everyone together in a Discord server and using that server to organize most writing processes. Writers and editors were able to freely collaborate and discuss questions, so the average question's life cycle was something like
- Writer writes question/comes up with a question idea
- Writer posts question/idea in server
- Editors and other writers give feedback on question/help complete an idea
- Writer revises question
- Editor revises question
- Cycle between 2-5 until final playtest and postediting [near set completion deadline]
karan
utexas
utexas
- Santa Claus
- Rikku
- Posts: 285
- Joined: Fri Aug 23, 2013 10:58 pm
Re: 2020 WORKSHOP: A Retrospective
Some context: both iterations of WORKSHOP primarily use/used comments on Google Docs to communicate between writers and editors, though this time around there's been more direct communication with editors through other channels as well. 2021 WORKSHOP also has a dedicated Discord server.
I would advocate for a set Discord as an alternative to sending out emails for announcements - it feels more personal and is definitely easier to handle and check as well. It made communication much simpler and, if you read my post, you know that's super important. If it can also serve other non-official purposes too, all the better.
I will push back slightly against the second half of this post though: I don't think such a server needs to be the place where editing happens. While there are certainly people out there with the skill set needed to edit multiple categories at any given difficulty, as you go up it's going to be hard for editors to give meaningful advice for questions outside their categories other than grammar or phrasing (which are, of course, important but also much simpler to spot and correct) so the benefits of this approach get a lot smaller. For lower difficulty sets where multiple people have the necessary expertise, though, this seems much more feasible.
It's also important to remember the power dynamics between writers and editors, especially for mentorship-oriented projects like WORKSHOP. I will immediately say that they're overblown and writers should always remember that their editors are humans and totally willing to listen to them, but nevertheless they exist in people's minds and can't be ignored. Some sets are going to be made up entirely of people who know each other and are thus comfortable, but I can easily imagine scenarios where writers or even other editors would be unwilling to step in to comment on questions being edited, or where people would feel intimidated to have their questions edited in a public space. I think having set Discords and writing documents separate is a reasonable middle ground that will work in most scenarios better, and having everyone chip in is something to aim for when possible.
Kevin Wang
Arcadia High School 2015
Amherst College 2019
2018 PACE NSC Champion
2019 PACE NSC Champion
Arcadia High School 2015
Amherst College 2019
2018 PACE NSC Champion
2019 PACE NSC Champion
Re: 2020 WORKSHOP: A Retrospective
You are correct that not many people have the knowledge and skill to effectively edit high-difficulty questions in terms of changing clues or conceits. However, I think many questions suffer from bad grammar, awkward phrasing, or confusing constructions, and that remedying those issues is not always simple. The more people who look at a question, the fewer of those defects that question has. Writing clear prose is an art that very few people are good at, but in my experience groups of people are fairly good at improving questions in a content-independent way.Santa Claus wrote: ↑Thu Nov 12, 2020 2:55 pm as you go up it's going to be hard for editors to give meaningful advice for questions outside their categories other than grammar or phrasing
I'm not sure what you mean here by "edited in a public space." If writers or editors don't feel comfortable doing group revision, where they bounce ideas off each other in real time, they are more than free to edit or rework questions in private and then post their revisions when done.Santa Claus wrote: ↑Thu Nov 12, 2020 2:55 pm I can easily imagine scenarios where writers or even other editors would be unwilling to step in to comment on questions being edited, or where people would feel intimidated to have their questions edited in a public space
Being able to directly converse with your mentors, as opposed to speaking indirectly through an opaque editing process, will likely help with power dynamics. Saturnalia had many new writers, and they quickly acclimated to collaborating with more established writers and editors because they were able to naturally converse and interact with their more experienced counterparts.Santa Claus wrote: ↑Thu Nov 12, 2020 2:55 pm It's also important to remember the power dynamics between writers and editors, especially for mentorship-oriented projects like WORKSHOP
karan
utexas
utexas
Re: 2020 WORKSHOP: A Retrospective
I'm obviously not Kevin, but it's important to keep in mind that asking for feedback on things you've put work into can be a very intimidating process for many people! I imagine a lot of people, especially newer writers who are more likely to be anxious about their abilities, would feel very uncomfortable with being required to post their questions for everyone to see and pick apart, compared to having a one-on-one connection with a single person. The approach you describe is one that I think could work well for sets that are exclusively written by experienced writers/editors, but I don't think it's a good idea when new writers are involved.entropy wrote: ↑Fri Nov 13, 2020 12:56 amI'm not sure what you mean here by "edited in a public space." If writers or editors don't feel comfortable doing group revision, where they bounce ideas off each other in real time, they are more than free to edit or rework questions in private and then post their revisions when done.Santa Claus wrote: ↑Thu Nov 12, 2020 2:55 pm I can easily imagine scenarios where writers or even other editors would be unwilling to step in to comment on questions being edited, or where people would feel intimidated to have their questions edited in a public space
Billy Busse
University of Illinois, B.S. '14
Rosalind Franklin University, M.S. '21, M.D. Candidate '25
Emeritus Member, ACF
Writer/Subject Editor/Set Editor, NAQT
University of Illinois, B.S. '14
Rosalind Franklin University, M.S. '21, M.D. Candidate '25
Emeritus Member, ACF
Writer/Subject Editor/Set Editor, NAQT
Re: 2020 WORKSHOP: A Retrospective
I'd like to thank Kevin for making this thread. I firmly believe more sets should have retrospective open discussion spaces for people to think and discuss them.
Looking back, 2020 WORKSHOP was a solid set that had some good ideas. It's certainly been better and more interesting to play than a lot of the sets I've played since then. It was unfortunate that it was advertised as being easier than it ended up being (which causes us to split our teams and playing the set on a shorthanded team was not the most fun thing in the world.) I apologize if my commentary ended up "poisoning the well" and I tried earnestly to give feedback to the writers in its discussion forum. As some might know, I take a lot of notes while playing and am probably an overly-critical person which is partially the reason I've largely stepped away from working on quizbowl things.
I think one good takeaway is not to take the "difficulty stars" as gospel or a hard and fast rule. It's pretty annoying to hear people say "that's not a 2.5 star tossup answerline!" or "man ,that's a rough bonus for 2 stars but might be more appropriate at 2.5 stars." This level of perfect gradation is not really possible and I worry that sometimes we get too married to these pseudo-quantitative measures. I tend to think of sets on an ordinal scale of "this is harder than that but easier than this other set" which I think prevents a lot of these mis-callibrations in difficulty. Tying yourself to the mast at a particular difficulty star (or worse, a projected average PPB or a PPB adjustment in high school) is overly ambitious. Perhaps most importantly, the difficulty scales are highly nonlinear and do not perfectly scale with the actual size of askable facts and answers.
Additionally, I'd be eager to see hear how this tournament's unique production schedule worked out. Was the set more on time than others or was it harder to corral due to the many writers? In my personal experience, the work on a quizbowl set tends to follow the Pareto principle closely but I'm curious if that was the case here.
Looking back, 2020 WORKSHOP was a solid set that had some good ideas. It's certainly been better and more interesting to play than a lot of the sets I've played since then. It was unfortunate that it was advertised as being easier than it ended up being (which causes us to split our teams and playing the set on a shorthanded team was not the most fun thing in the world.) I apologize if my commentary ended up "poisoning the well" and I tried earnestly to give feedback to the writers in its discussion forum. As some might know, I take a lot of notes while playing and am probably an overly-critical person which is partially the reason I've largely stepped away from working on quizbowl things.
I think one good takeaway is not to take the "difficulty stars" as gospel or a hard and fast rule. It's pretty annoying to hear people say "that's not a 2.5 star tossup answerline!" or "man ,that's a rough bonus for 2 stars but might be more appropriate at 2.5 stars." This level of perfect gradation is not really possible and I worry that sometimes we get too married to these pseudo-quantitative measures. I tend to think of sets on an ordinal scale of "this is harder than that but easier than this other set" which I think prevents a lot of these mis-callibrations in difficulty. Tying yourself to the mast at a particular difficulty star (or worse, a projected average PPB or a PPB adjustment in high school) is overly ambitious. Perhaps most importantly, the difficulty scales are highly nonlinear and do not perfectly scale with the actual size of askable facts and answers.
Additionally, I'd be eager to see hear how this tournament's unique production schedule worked out. Was the set more on time than others or was it harder to corral due to the many writers? In my personal experience, the work on a quizbowl set tends to follow the Pareto principle closely but I'm curious if that was the case here.
Harris Bunker
Grosse Pointe North High School '15
Michigan State University '19
UC San Diego Economics 2019 -
at least semi-retired
Grosse Pointe North High School '15
Michigan State University '19
UC San Diego Economics 2019 -
at least semi-retired
- Santa Claus
- Rikku
- Posts: 285
- Joined: Fri Aug 23, 2013 10:58 pm
Re: 2020 WORKSHOP: A Retrospective
I think it ended up pretty similarly to any other set, which is a pretty good result considering the additional pressures on the editors. It came down to the last hours of the last day, but I heard of very few sets that haven't had that happen.
EDIT: Forgot to include some thoughts.
I've already mentioned how, for 2021 WORKSHOP, we shifted from divvying up the entire distribution between writers from the start to a chunk-based system to prevent people from overcommitting early on, and wanted to approach that again from the lens of efficiency. We experienced many delays because we did not want to rearrange these divisions once we had set them, for fear of stepping on people's toes - this was compounded by our reluctance to rigidly enforce our deadlines. You can't really have both of these things - you either can have soft deadlines and a similarly amorphous system of determining who writes what or you can create rigid assignments and equally rigid deadlines. It'll take all of five seconds to realize why the latter won't work, so I would really promote the former - our more modular approach for 2021 WORKSHOP has meant that we could respond much more quickly to writers telling us they wanted to write more or write less, and we've totally avoided the problem of getting stuck at this purely logistical stage.
If you flip through the packets of WORKSHOP, you'll see the same writer tags pop up a lot, yeah, but I don't really mind it too much. I think in general the community really focuses on the superstars who sit there and churn out questions; obviously you need some people to do this sort of thing, since sets don't get finished unless people write for them, but part of what I've tried to emphasize more in WORKSHOP this year is that writers don't need to write a lot to be a good writer, or to be an important contribution to a set. People write at different speeds and with different breadth and there shouldn't be any shame in saying upfront, "Hey, I'm going to write 4/4", putting in your best effort, and then stopping there.was it harder to corral due to the many writers? In my personal experience, the work on a quizbowl set tends to follow the Pareto principle closely but I'm curious if that was the case here.
Kevin Wang
Arcadia High School 2015
Amherst College 2019
2018 PACE NSC Champion
2019 PACE NSC Champion
Arcadia High School 2015
Amherst College 2019
2018 PACE NSC Champion
2019 PACE NSC Champion
-
- Yuna
- Posts: 973
- Joined: Tue Aug 02, 2016 12:31 pm
- Location: Indiana
Re: 2020 WORKSHOP: A Retrospective
Regarding the schedule, I recall most of the last-minute work at least on my end coming in the form of filling in question slots that were originally taken by people who flaked or wrote questions that I considered beyond my power to fix (a lot of this was certainly down to my habit of editing the easier-to-edit questions first). I don't think I really succeeded in establishing a real mentoring relationship with every writer in my category, which, as above, I'm not sure whether to ascribe to any personal failing or to the conceit of the set itself. I did see marked improvement in a lot of writers that I was able to work more closely with, though, which indicates to me that the set largely accomplished its goal. I do consider our scheduling guidelines a failure, though: as editors (myself included) we didn't really do a good job of adhering to them, which probably made it harder for writers to take them seriously. I would say to anyone contemplating a similar project that consistency is probably the most important practice for editing something like this.
Jakob M. (they/them)
Michigan State '21, Indiana '2?
"No one has ever organized a greater effort to get people interested in pretending to play quiz bowl"
-Ankit Aggarwal
Michigan State '21, Indiana '2?
"No one has ever organized a greater effort to get people interested in pretending to play quiz bowl"
-Ankit Aggarwal