2020 WORKSHOP: A Retrospective

Old college threads.
Locked
User avatar
Santa Claus
Rikku
Posts: 285
Joined: Fri Aug 23, 2013 10:58 pm

2020 WORKSHOP: A Retrospective

Post by Santa Claus »

Last year I edited biology, other science, and myths/legends for 2020 WORKSHOP, a set intended to be an opportunity for newer writers to write questions with the aid of mentorship as a spiritual successor to the PADAWAN set. This year, I am reprising my editing roles for 2021 WORKSHOP, as well as serving as co-head editor alongside Olivia Murton and Wonyoung Jang. I am writing this retrospective in an effort to expand the benefits of this project to the wider community and be transparent about the success and failures of our work.

Throughout this post I will frequently use “the royal we” so I can pretend that I am speaking for anyone other than myself - others involved in the process are encouraged to add their voices into the mix.

What didn’t work

I’m opening with this section to immediately address some of the pitfalls we ourselves got stuck in. Some of these were unique to our situation - some weren’t.

We did not do as good of a job of being mentors as we could have

I will take a moment to split the hair between “mentor” and “editor” for a project in which everyone was both - as editors our job was to take in raw questions and produce a final set, and as evinced by the fact that many people played 2020 WORKSHOP we succeeded in that goal. As mentors our job was to provide the best space we could for our mentees to try writing for the first time and we did not do as well on that front.

In some sense, this was unavoidable. Everyone is human, and it’s hard to sustain the same level of engagement with a project for its entirety, especially as mentorship is very taxing. I am not criticizing any of my fellow editors on either 2020 or 2021 WORKSHOP, as they all put in hard work and got good results, but there is no point in sugarcoating the fact that as a whole we could have done better.

The biggest problem was the speed of feedback. As part of the production pipeline (referred to internally as the “WORKSHOP Workflow”), writers needed to receive approval for answerlines, and would typically receive feedback that would allow them to self-edit questions before a final pass by the editor (I am aware that there are other projects that skip this first step and allow writers to produce full tossups before requiring editor oversight, but I think that both approaches have their upsides). When editors took time (days, weeks, sometimes months) to respond to proposed answerlines, or further drafts, that really hampered the ability of writers to participate in the process and stripped them of some of their agency as writers.

There is no real solution to this, but communication with editors is the biggest tool to avoid this and establish accountability. For 2021 WORKSHOP, we have stressed to our writers that they can talk to editors and even ask them to edit their questions - this should be the norm for all sets. Additionally, I did not consider myself in a position to ask my fellow editors to do things last year, and I think this is not uncommon - it is very difficult to tell people you consider your peers or even your betters that they haven’t done their job. Head editors have to take the initiative on this sort of thing.

Our organizational system was not good

2020 WORKSHOP had a very complicated system for keeping track of how much people had written and incentivizing writers to both write more and write consistently - I am very much in the debt of Will Grossman for his work in making the cogs turn. It involved tracking how much each writer had written each week and adjusting the amount they were “required” (read: asked politely) to write for the next week on the fly, with monetary bonuses given to writers who were able to meet these goals every week. It was not good - it had too many moving parts and the only value was the ability to award consistency bonuses, which unduly punishes writers who write less or write slower in favor of writers who already are getting compensated more by virtue of writing more.

For 2021 WORKSHOP, we have moved towards a more straight-forward “chunk”-based system, in which each writer chooses 4/4 to write and they can write additional chunks as they finish the previous one, and have entirely removed consistency bonuses - I believe we took inspiration from Harvard’s system when writing HFT, and I don’t doubt that other sets have done this as well. This significantly reduced day-to-day overhead and I would recommend it for this reason, as well as some I will discuss in the next section.

Some of our writers did not write and we were not proactive about our solutions

Writers saying they can write some number of questions and ultimately not doing so is a very common scenario, especially for a project like WORKSHOP where many writers are taking a step out into the unknown and committing to something they may not have done extensively before. In general though, editors and writers have no reason to worry about individuals not producing as many questions as they claimed they would, so long as the set is finished and no one is unduly inconvenienced - others will simply pick up the slack.

The problem was that we were split between being editors working towards completing a set and mentors who were trying to ensure the maximum opportunity possible for all of our writers. As part of this second goal, we had allowed each writer to explicitly state how many questions they wanted to write in their application and had split up the distribution accordingly. When a writer did not meet deadlines, we could not impel them to write in the way that one could force a more experienced writer or a teammate to do so. We could not take the other approach, either, and open their section of the distro up to other writers, because we could not meaningfully decide which questions they still had the potential to write and removing their entire claim would prevent them from writing at all. So we sat on our hands, and had to deal with the consequences as the first mirror approached.

The first solution to this was yet again communication. In this iteration of WORKSHOP we have tried much harder to be in contact with individual writers and this means that we have not needed to guess at what their intentions were - they would simply tell us. The second was to switch to the chunk system mentioned earlier, which meant that no writer could overcommit at the beginning and it was much easier to dynamically change the expectations as the writing process went on.

We overshot difficulty

Ignoring for a moment the fact that the set was accidentally announced to be half a step lower in difficulty than intended, 2020 WORKSHOP was harder than the projected difficulty. Power counts across mirrors were consistently depressed, and had we had Advanced Stats or other forms of buzzpoint analysis we would have almost certainly seen many clues unbuzzed on.

Not much we could do here but adjust expectations appropriately - both ours and the audience’s. We are more consciously writing questions to be easier for 2021 WORKSHOP, but people reading this post shouldn’t be too surprised in a year’s time when we’ve accidentally overshot our target again - everyone does it.

What did work

We were able to provide mentorship for many writers

I am very happy to see that many of the writers who worked on 2020 WORKSHOP have continued on to other projects, and I’m keeping an eye out for those that I haven’t seen yet. Despite the aforementioned inefficiencies in our process, the vast majority of the questions were written by our writer core, which is quite a feat when you consider their overall inexperience and the difficulty level of the set - it hasn’t happened since PADAWAN! 2021 WORKSHOP has fixed many of the problems of the previous year and ideally will have success as well, and I can only hope that WORKSHOP becomes a long-standing institution in the quiz bowl community.

There’s been a lot of discussion in the community recently about the role of mentorship programs, especially when compared to packet sub. I don’t think there can be any doubt that dedicated projects like WORKSHOP serve an important role, but I will briefly reiterate my stance here that both those and other avenues are necessary to reach different kinds of people.

Having non-editor people on the team is amazing

I think, sitting in the editor’s chair and looking back, the best part of 2020 WORKSHOP was having dedicated people for tasks other than editing - proofreading, logistics, mirror organization, etc. For 2021 WORKSHOP we expanded the program to include mentoring opportunities for people who want to do logistics work in quiz bowl, which means that we have an entire team of people for getting mirrors organized. Anyone who has tried to juggle editing duties with proofreading or weighing prospective hosts will know how hard it can be, so all I can say is: get yourself a logistics team, even if it’s just one person.

The spreadsheets we used for answerlines were very powerful

Any project written by more than a few people needs to have a plan for how to track things like the subdistribution, question answerlines, what writers have written, what people are owed, how much the set is written, etc. This was especially true for WORKSHOP, which had six editors and 42 writers. We were able to have all of this information integrated into a single spreadsheet, which would automatically keep track of who had written what by annotating answers in the distribution with writer and editor tags. The incredible efficiency of this document allowed editors and writers alike to just glance at a single place to figure out what tasks they had left to do or see the overall progress of the set.

If anyone would be interested in using this sheet for their projects, please contact me or one of the other head editors of WORKSHOP and we will gladly work to adapt the system to your needs.

Our points-based compensation system was very streamlined

Rather than assigning every question a dollar amount at the get-go, we made use of a point system that allowed us to divy up set profits between writers and editors after all mirrors were finished. Each tossup was worth 5 points in total and each bonus 4 - when a question was edited, three points would go to the editor and the remainder to the writer, meaning 2 for tossups and 1 for bonuses. A simplified version of this was also used for proof-reading work, though that was primarily a count of questions checked.

One feature that we decided to remove between 2020 and 2021 was the “minimal editing bonus”, which was a flag that indicated that a question had meant some standard of quality and thus one additional point would be given to the writer and one fewer to the editor. There was no way to ensure consistency in how this bonus was awarded between editors and in general it felt too subjective a measure for us to feel comfortable using it to determine payment.

The set was good

Difficulty concerns aside, I think 2020 WORKSHOP was a very good set; I was personally pleased to see specific praise of the science as someone who was responsible for roughly half of it. It is unfortunate that so few teams got to play it because of the abrupt shortening of the 2019-2020 season, but it is my hope that teams will be able to appreciate the ideas and the clues as they study the set in years to come.
Kevin Wang
Arcadia High School 2015
Amherst College 2019

2018 PACE NSC Champion
2019 PACE NSC Champion
User avatar
entropy
Wakka
Posts: 107
Joined: Sun Feb 11, 2018 5:03 pm

Re: 2020 WORKSHOP: A Retrospective

Post by entropy »

Santa Claus wrote: Thu Nov 12, 2020 12:36 pm communication
Kevin said in the main Discord that 2020 WORKSHOP didn't have a writing Discord server, which I think is something every set [or at least every set with new writers] really needs. CALISTO and Saturnalia facillitated inter-writer/editor communication by keeping everyone together in a Discord server and using that server to organize most writing processes. Writers and editors were able to freely collaborate and discuss questions, so the average question's life cycle was something like
  1. Writer writes question/comes up with a question idea
  2. Writer posts question/idea in server
  3. Editors and other writers give feedback on question/help complete an idea
  4. Writer revises question
  5. Editor revises question
  6. Cycle between 2-5 until final playtest and postediting [near set completion deadline]
This process improves set quality because it means that each question is subject to huge amounts of scrutiny [far more than there would be if questions went between one writer and one/two editors] and that writers and editors are able to naturally collaborate with each other to create a coherent set philosophy. I think the latter point is especially important, because it means new writers are surrounded by stronger writers and are able to quickly acquire good question-writing values and experience in revising questions.
karan
utexas
User avatar
Santa Claus
Rikku
Posts: 285
Joined: Fri Aug 23, 2013 10:58 pm

Re: 2020 WORKSHOP: A Retrospective

Post by Santa Claus »

entropy wrote: Thu Nov 12, 2020 2:05 pmusing that [writing Discord server] to organize most writing processes
Some context: both iterations of WORKSHOP primarily use/used comments on Google Docs to communicate between writers and editors, though this time around there's been more direct communication with editors through other channels as well. 2021 WORKSHOP also has a dedicated Discord server.

I would advocate for a set Discord as an alternative to sending out emails for announcements - it feels more personal and is definitely easier to handle and check as well. It made communication much simpler and, if you read my post, you know that's super important. If it can also serve other non-official purposes too, all the better.

I will push back slightly against the second half of this post though: I don't think such a server needs to be the place where editing happens. While there are certainly people out there with the skill set needed to edit multiple categories at any given difficulty, as you go up it's going to be hard for editors to give meaningful advice for questions outside their categories other than grammar or phrasing (which are, of course, important but also much simpler to spot and correct) so the benefits of this approach get a lot smaller. For lower difficulty sets where multiple people have the necessary expertise, though, this seems much more feasible.

It's also important to remember the power dynamics between writers and editors, especially for mentorship-oriented projects like WORKSHOP. I will immediately say that they're overblown and writers should always remember that their editors are humans and totally willing to listen to them, but nevertheless they exist in people's minds and can't be ignored. Some sets are going to be made up entirely of people who know each other and are thus comfortable, but I can easily imagine scenarios where writers or even other editors would be unwilling to step in to comment on questions being edited, or where people would feel intimidated to have their questions edited in a public space. I think having set Discords and writing documents separate is a reasonable middle ground that will work in most scenarios better, and having everyone chip in is something to aim for when possible.
Kevin Wang
Arcadia High School 2015
Amherst College 2019

2018 PACE NSC Champion
2019 PACE NSC Champion
User avatar
entropy
Wakka
Posts: 107
Joined: Sun Feb 11, 2018 5:03 pm

Re: 2020 WORKSHOP: A Retrospective

Post by entropy »

Santa Claus wrote: Thu Nov 12, 2020 2:55 pm as you go up it's going to be hard for editors to give meaningful advice for questions outside their categories other than grammar or phrasing
You are correct that not many people have the knowledge and skill to effectively edit high-difficulty questions in terms of changing clues or conceits. However, I think many questions suffer from bad grammar, awkward phrasing, or confusing constructions, and that remedying those issues is not always simple. The more people who look at a question, the fewer of those defects that question has. Writing clear prose is an art that very few people are good at, but in my experience groups of people are fairly good at improving questions in a content-independent way.
Santa Claus wrote: Thu Nov 12, 2020 2:55 pm I can easily imagine scenarios where writers or even other editors would be unwilling to step in to comment on questions being edited, or where people would feel intimidated to have their questions edited in a public space
I'm not sure what you mean here by "edited in a public space." If writers or editors don't feel comfortable doing group revision, where they bounce ideas off each other in real time, they are more than free to edit or rework questions in private and then post their revisions when done.
Santa Claus wrote: Thu Nov 12, 2020 2:55 pm It's also important to remember the power dynamics between writers and editors, especially for mentorship-oriented projects like WORKSHOP
Being able to directly converse with your mentors, as opposed to speaking indirectly through an opaque editing process, will likely help with power dynamics. Saturnalia had many new writers, and they quickly acclimated to collaborating with more established writers and editors because they were able to naturally converse and interact with their more experienced counterparts.
karan
utexas
touchpack
Rikku
Posts: 450
Joined: Tue Sep 20, 2011 12:25 am

Re: 2020 WORKSHOP: A Retrospective

Post by touchpack »

entropy wrote: Fri Nov 13, 2020 12:56 am
Santa Claus wrote: Thu Nov 12, 2020 2:55 pm I can easily imagine scenarios where writers or even other editors would be unwilling to step in to comment on questions being edited, or where people would feel intimidated to have their questions edited in a public space
I'm not sure what you mean here by "edited in a public space." If writers or editors don't feel comfortable doing group revision, where they bounce ideas off each other in real time, they are more than free to edit or rework questions in private and then post their revisions when done.
I'm obviously not Kevin, but it's important to keep in mind that asking for feedback on things you've put work into can be a very intimidating process for many people! I imagine a lot of people, especially newer writers who are more likely to be anxious about their abilities, would feel very uncomfortable with being required to post their questions for everyone to see and pick apart, compared to having a one-on-one connection with a single person. The approach you describe is one that I think could work well for sets that are exclusively written by experienced writers/editors, but I don't think it's a good idea when new writers are involved.
Billy Busse
University of Illinois, B.S. '14
Rosalind Franklin University, M.S. '21, M.D. Candidate '25
Emeritus Member, ACF
Writer/Subject Editor/Set Editor, NAQT
User avatar
Ciorwrong
Tidus
Posts: 696
Joined: Fri Dec 20, 2013 8:24 pm

Re: 2020 WORKSHOP: A Retrospective

Post by Ciorwrong »

I'd like to thank Kevin for making this thread. I firmly believe more sets should have retrospective open discussion spaces for people to think and discuss them.

Looking back, 2020 WORKSHOP was a solid set that had some good ideas. It's certainly been better and more interesting to play than a lot of the sets I've played since then. It was unfortunate that it was advertised as being easier than it ended up being (which causes us to split our teams and playing the set on a shorthanded team was not the most fun thing in the world.) I apologize if my commentary ended up "poisoning the well" and I tried earnestly to give feedback to the writers in its discussion forum. As some might know, I take a lot of notes while playing and am probably an overly-critical person which is partially the reason I've largely stepped away from working on quizbowl things.

I think one good takeaway is not to take the "difficulty stars" as gospel or a hard and fast rule. It's pretty annoying to hear people say "that's not a 2.5 star tossup answerline!" or "man ,that's a rough bonus for 2 stars but might be more appropriate at 2.5 stars." This level of perfect gradation is not really possible and I worry that sometimes we get too married to these pseudo-quantitative measures. I tend to think of sets on an ordinal scale of "this is harder than that but easier than this other set" which I think prevents a lot of these mis-callibrations in difficulty. Tying yourself to the mast at a particular difficulty star (or worse, a projected average PPB or a PPB adjustment in high school) is overly ambitious. Perhaps most importantly, the difficulty scales are highly nonlinear and do not perfectly scale with the actual size of askable facts and answers.

Additionally, I'd be eager to see hear how this tournament's unique production schedule worked out. Was the set more on time than others or was it harder to corral due to the many writers? In my personal experience, the work on a quizbowl set tends to follow the Pareto principle closely but I'm curious if that was the case here.
Harris Bunker
Grosse Pointe North High School '15
Michigan State University '19
UC San Diego Economics 2019 -

at least semi-retired
User avatar
Santa Claus
Rikku
Posts: 285
Joined: Fri Aug 23, 2013 10:58 pm

Re: 2020 WORKSHOP: A Retrospective

Post by Santa Claus »

Ciorwrong wrote: Fri Nov 13, 2020 5:33 pm Additionally, I'd be eager to see hear how this tournament's unique production schedule worked out. Was the set more on time than others,
I think it ended up pretty similarly to any other set, which is a pretty good result considering the additional pressures on the editors. It came down to the last hours of the last day, but I heard of very few sets that haven't had that happen.

EDIT: Forgot to include some thoughts.

I've already mentioned how, for 2021 WORKSHOP, we shifted from divvying up the entire distribution between writers from the start to a chunk-based system to prevent people from overcommitting early on, and wanted to approach that again from the lens of efficiency. We experienced many delays because we did not want to rearrange these divisions once we had set them, for fear of stepping on people's toes - this was compounded by our reluctance to rigidly enforce our deadlines. You can't really have both of these things - you either can have soft deadlines and a similarly amorphous system of determining who writes what or you can create rigid assignments and equally rigid deadlines. It'll take all of five seconds to realize why the latter won't work, so I would really promote the former - our more modular approach for 2021 WORKSHOP has meant that we could respond much more quickly to writers telling us they wanted to write more or write less, and we've totally avoided the problem of getting stuck at this purely logistical stage.
was it harder to corral due to the many writers? In my personal experience, the work on a quizbowl set tends to follow the Pareto principle closely but I'm curious if that was the case here.
If you flip through the packets of WORKSHOP, you'll see the same writer tags pop up a lot, yeah, but I don't really mind it too much. I think in general the community really focuses on the superstars who sit there and churn out questions; obviously you need some people to do this sort of thing, since sets don't get finished unless people write for them, but part of what I've tried to emphasize more in WORKSHOP this year is that writers don't need to write a lot to be a good writer, or to be an important contribution to a set. People write at different speeds and with different breadth and there shouldn't be any shame in saying upfront, "Hey, I'm going to write 4/4", putting in your best effort, and then stopping there.
Kevin Wang
Arcadia High School 2015
Amherst College 2019

2018 PACE NSC Champion
2019 PACE NSC Champion
A Dim-Witted Saboteur
Yuna
Posts: 973
Joined: Tue Aug 02, 2016 12:31 pm
Location: Indiana

Re: 2020 WORKSHOP: A Retrospective

Post by A Dim-Witted Saboteur »

Regarding the schedule, I recall most of the last-minute work at least on my end coming in the form of filling in question slots that were originally taken by people who flaked or wrote questions that I considered beyond my power to fix (a lot of this was certainly down to my habit of editing the easier-to-edit questions first). I don't think I really succeeded in establishing a real mentoring relationship with every writer in my category, which, as above, I'm not sure whether to ascribe to any personal failing or to the conceit of the set itself. I did see marked improvement in a lot of writers that I was able to work more closely with, though, which indicates to me that the set largely accomplished its goal. I do consider our scheduling guidelines a failure, though: as editors (myself included) we didn't really do a good job of adhering to them, which probably made it harder for writers to take them seriously. I would say to anyone contemplating a similar project that consistency is probably the most important practice for editing something like this.
Jakob M. (they/them)
Michigan State '21, Indiana '2?
"No one has ever organized a greater effort to get people interested in pretending to play quiz bowl"
-Ankit Aggarwal
Locked