2020 SCT general discussion

[Important Bird Area]

This is your discussion thread for big-picture issues about the 2020 NAQT SCT (either division). If you'd like to discuss the specific text of a particular question, please use the two threads available for that purpose.

[VSCOelasticity]

The science of this set was fine, from what I could tell while playing. There were occasional cliffs and some lead-ins that seemed a little soft for DI SCT, but that happens sometimes and nothing felt egregious.

Outside of that, the most annoying thing about this tournament was the terrible subdistributing. There was way too much opera and baseball, at the expense of jazz/folk/other and football/basketball. I think I remember two jazz bonuses? I think there were zero NBA question throughout the day, and only one NFL question. We only played 9 rounds, but still.

Overall, this set felt like it could've used another few days of content and copy editing.

[Cheynem]

I calculated the sports questions:

Baseball
3/4 (packets 2, 3, 6, 8, 9, 13, 14)

College Basketball
2/0 (packets 11, 12)

College Football
0/1 (packet 4)

Golf
1/0 (packet 15)

Misc.
1/1 (tossup included UFC and NHL, packet 5; bonus included NFL and college basketball, packet 7)

NBA
0/1 (packet 16)

NFL
1/1 (packets 1, 10)

What seemed to happen is that while baseball, football, and basketball take up about the same space, the space for "Any Sports" ended up mostly going to baseball, and college sports (especially basketball) siphoned off space for the NBA and NFL. Furthermore, almost all of the baseball questions were dumped early in the set, so that added to the feeling even more.

A lot of this should probably be caught in subdistro checking, of course, and this may have just unfortunately fallen through the cracks--there's no way that there should be more college basketball questions than the NBA, and the "Any Sports" section should be more evenly distributed. Some of this is on the editors and some of this is on the writers.

Anyway, as the writer of a fair share of those questions, I hope at least they were good!

[Important Bird Area]

way too much opera

Opera was our standard 5/5 per SCT set (plus one more tossup in the "Any Fine Arts" section). All of the tossups were in the first 11 rounds, so it may have felt more prevalent than usual.

[vinteuil]

I don’t think it would be justified for me to criticize the overall editing of this set, because, as far as I could tell, there was no set editing, overall.

I don’t mean “the editors of this set did nothing.” I’m sure the subject editors did their job of making sure clues were buzzable, correct, and pyramidal (the science was by and large quite good), and I’m sure that the set editors made plenty of changes: correcting typos and ensuring grammaticality—the fundamentals of playability. (Although a lack of prompt on “bar” for the ABA and a bonus part on Shintoism right after question text mentioning “in Shinto temples” indicate that even these basics were not being executed consistently.)

Rather, I mean that zero effort seems to have been taken to ensure that the questions actually do their job of discriminating between teams trying to qualify for a national championship. Broadly speaking, this manifested as an utter lack of consistency. A fuck-you parliamentary procedure bonus after a Holbein tossup dropping one of his most famous paintings in the leadin. Guaranteed buzzer races on first clues like “minimal ___ paradigm” or Le Loi and the turtle. Tossups with one substantive clue in power (e.g. “Ain’t I a Woman”), or that cliff directly from the Pauline Kael quote to famous names from the Saturday Night Massacre are not built to distinguish teams. And (relatively difficulty-flat) tossups on Leontyne Price and La Forza del Destino, or on Letters from a Pennsylvania Farmer, should not be coexisting with bonuses where the purported hard part is The Nun’s Priest or Aurelian (with all the clues in both cases).

What was the philosophy behind easy parts and giveaways? There wasn’t one, as exemplified by the contrast between the “rebels” and “sneezing” giveaways and the easy part on “soccer.” (Which, in turn, should be compared to the noticeably more difficult easy parts on Al Sharpton and Max Payne.)

In sum, this was the least consistent all-subject set I’ve played since St. Mary’s high school’s annual housewrite in 2011.

I don’t want to end here, though. I think this set’s failures can be directly attributed to a lack of oversight within NAQT’s editorial system. Let me start by laying out some of the details of that system.

Subject editors are supposed to ensure that the questions they edit are accurate, pyramidal, well-formatted, and coded for an appropriate range of difficulties. They edit questions over a very long period of time and should not be expected to make difficulty entirely consistent. Indeed, NAQT rightly likes to maintain some flexibility by having many questions be coded for multiple difficulties, with the understanding that the question (certainly the answerline) is roughly appropriate for those levels. This practice rests on the understanding that set editors will edit the set, making difficulty consistent.

There is no particular set of guidelines for what set editors (FEED and SEED) should do beyond ensuring that questions are packetized correctly, factually correct, and typo-free. Nonetheless, it is understood that those entrusted with these roles will shape the set into a consistent whole. I remember the two SCTs that Stephen, Auroni, and I edited. It took a HUGE amount of work to make the middle clues and bonus parts consistent. But there was nobody making sure we would do that; with so many editors involved, I often found that I had to snap myself out of the “somebody else will deal with this, it’s so obvious” mentality. And unlike with other sets, the editors are not necessarily advertised, and take a back seat to “NAQT” when it comes time to talk about the set’s successes and failures.

It might be worth asking if a set editors should be compensated more for this work; I made about $500 for SEED in 2017 and 2018, which isn’t great for the amount of work—although remember that the most labor-intensive tasks of subject editing and writing are already done. In any case, I’m not sure what NAQT can really do to ensure that their set editors make difficulty consistent.

Which brings me to a quandary. Look, I’ve been as consistent an advocate for NAQT as anyone, and I’m extremely satisfied as a former employee. I enjoy NAQT’s product when well-executed, and sets like the 2017 ICT are among my favorites. But given the slip-ups of the 2019 ICT and the catastrophes of this set—and, more saliently, the lack of any real way to guarantee quality control—I am extremely wary about playing future NAQT sets. I do not plan to play another SCT until NAQT has built a track record of producing quality college sets (e.g. a successful ICT this year and SCT next year); and if ICT continues its downward trajectory, I plan to boycott NAQT entirely and encourage others to do the same.

[Here Comes Rusev Day]

I staffed the Florida site of this and thought the set was indeed "inconsistent," but by no means was it a crime against quizbowl or even abysmal for that matter. I mostly read DII, but there were definitely some questions which seemed straight out of 2007 (boreholes, a strange Greek letters in science bonus), DI questions that were downgraded and just had the lead-in taken out for DII (leading to as mentioned inconsistent difficulty), and probably the worst sub distribution/packetization I've ever seen in any set. I always greatly appreciated that NAQT sets are unique from standard ACF and college housewrite fare, but how "different" these sets should be is a question we should start asking. With that said, I know this set will be smoothed out for CCCT, because those players deserve to play as quality a set as possible for their national championship.

[Cheynem]

I'm not particularly convinced the "uniqueness" of NAQT is really the main issue here--there have been well-received SCTs and ICTs using the same NAQT distro and format, and most of the issues people have about this particular set seem to be like difficulty inconsistency/packetization/sub distro issues that aren't really connected to how "different" NAQT is from other sets (you can, I suppose, believe as Jacob does that the NAQT style of producing sets can result in these issues).

[Jack]

I want to echo the comment about too much baseball: There was too much baseball. Although it's true that baseball is still probably the second-most popular sport in the US, it's also true that it's more popular with older people (e.g. people not of quiz bowl age). If you don't actually like sports (and, for whatever reason, wanted to 'study' sports for quiz bowl), but follow sportscenter or ESPN on social media, like on Instagram or Facebook, the NFL and NBA (and probably also college football and basketball) are significantly more covered/culturally significant than anyone or any team in the MLB. LeBron James and Tom Brady are household names, but there's no MLB equivalent (Mike Trout?). Having a bonus about three good MLB players whose fathers also played in the MLB, for example, is really just a test for baseball fans, and not really of more general 'pop culturey' knowledge these days.

Not that this is really all that important, anyway, since it's just trash. I guess it's somewhat worth mentioning since having more pop culture is NAQT's "shtick," so having it done well is always preferential. The bigger issue with this year's SCT, though, was the inconsistent bonus difficulty, which I'm sure others will likely discuss at length.

[vinteuil]

I should say that, in laying the blame on the set editors above, I was working from the assumption that the subject editors did their jobs well. If they didn't, that creates far more work than you could possibly expect two or three set editors to handle. (Of course, the problem of editorial "oversight" is the same, just applying at a different level.)

I also hope I didn't sound too pessimistic at the end. I believe in NAQT's ability to improve its process (or, in a select few cases, reduce or eliminate the role of offending parties), and I have deep respect for many of its employees as writers and editors (including, or even especially, Andrew, Rob, and Billy). I believe that ICT and other future NAQT college tournaments can be truly great, and I certainly don't hope to take action against the company; I'd much prefer to be able to enjoy its unique format, and I look forward to hearing about efforts to ensure future quality control.

[Cheynem]

I don't think anyone would really disagree with Jack regarding baseball--I don't think this was like an ideological decision to value baseball more than the other sports (it certainly wasn't on my end). It was just a (very unfortunate) error regarding sub-distributions--I don't offer this is an excuse but rather an attempt to reassure folks that it wasn't like NAQT actually believes that baseball should fundamentally have that much room in a distribution.

[Jack]

I don't think anyone would really disagree with Jack regarding baseball--I don't think this was like an ideological decision to value baseball more than the other sports (it certainly wasn't on my end). It was just a (very unfortunate) error regarding sub-distributions--I don't offer this is an excuse but rather an attempt to reassure folks that it wasn't like NAQT actually believes that baseball should fundamentally have that much room in a distribution.

Good to hear. Things can happen, and it's just unfortunate it happened this time.

[Sigurd]

Similar to some complaints raised here about different categories, but my team was quite strongly off the opinion that the literature in this set was very heavily weighted towards drama/plays, and I did get that impression myself (as a staffer). I see that the literature is not subdivided by genre (only by geography/religion), so I understand if a exact response is not possible, but I'm wondering if anyone else felt this way or if NAQT considers the genre of literature when putting together their sets.

[naan/steak-holding toll]

I'd like to start with several positive notes:

• The current events questions were a notable improvement from last year, and featured many fewer questions about "Trumpbowl," minor officials, scandals, and tweets
• The geography questions were consistently enjoyable, interesting, and difficulty appropriate
• The science, apart from the extremely bad machine learning tossup and a few pretty easy tossups (rings, rotation), was generally quite good

I completely agree with Jacob on most of the points he has outlined. I think my biggest source of disappointment with this tournament's inconsistency and lack of quality control is the fact that the set editors are some of the best editors we have in the community and have produced many very successful tournaments before.

Additional subdistributional notes for the 12 rounds we played:

• In addition to five opera questions, there was also a tossup on Leontyne Price (an opera singer), which made the lack of jazz stick out more
• The literature tossups were consistently hard to power - you had one to know alone specific short story by Sholokhov to power that tossup, to cite one example
• The non-U.S. history tossups were overwhelmingly on political and military history, to the general exclusion of other areas
• The trash bonuses were really, really hard - not just for the general audience, but strong trash players as well

At the risk of sounding like a cliche business student, I further agree with Jacob that process improvement could probably go a long way towards solving some of these issues - making sure that not everything falls through the cracks is very challenging. I'll spend some time thinking about how this could be done and maybe write up something about it in the future.

EDIT: ADDENDUM ON A DIFFERENT NOTE

I very much appreciate the fact that NAQT continues to appeal to a broad range of people by including general knowledge, geography, pop culture, and a more fast-paced gameplay style; I think these are features, not bugs, and the precedent of strong sets such as the 2014-16 SCTs and 2016-18 ICTs suggest that the organization is very capable of producing tournaments that have a wide range of interesting content.

That said, I think the experiences I observed for both strong and weak teams at this weekend's SCT suggest that the current timing rules are frustrating and disliked by a substantial majority of players. There are several reasons:

• The percent of players who are punished by the shrinking of timing (compared to ACF rules) is much higher. Simply, if you give people fewer seconds to pull an answer when buzzing in, then they're much more likely to get a neg when they know the correct answer and "deserve" points. In a fast-paced game format like NAQT, I think you definitely get higher stress levels, which can make it even more challenging to pull things quickly - so you get a doubling of this frustration. It's especially discouraging to see new players buzz in and not be able to remember that quickly.
• Players with, for lack of a better term, "technical knowledge" of subjects are further disproportionately punished. NAQT tossups, like other good question sets these days, have a number of "technical" clues in music and science (equations, score clues, etc.) that can take a few seconds to parse - a player who recognizes a clue thus faces a really big risk when buzzing immediately that they won't have time to think through and give a correct answer. This was made doubly worse by several descriptive clues being outright non-unique, causing more buzz hesitancy.
• Not the fault of the rules per se, but rather of there only being two college NAQT events per year: Moderators are not very used to enforcing these rules, so it naturally leads to some variance - some mods will hesitate a bit and end up giving you three seconds, others will be super strict and fast. This inconsistency also frustrates the player experience.

I think it's possible to compromise and maybe change to 3-second timing, taking a midpoint between the current fast paced-format and the slower format of mACF. This also has the aesthetic advantage of giving a more consistent set of timing rules with 3 seconds after buzzing, 3 seconds at the end of the tossup, 3 seconds on bonus prompts (the exception is 5 seconds on bonuses, which I think is generally agreed to be something that you can't change). It should also be plenty possible to implement with the new 11-minute timing rule, under which a good majority of games at the ICT level finish more than 20 tossups per packet already.

[gerbilownage]

I wanted to agree with Isaac and note the haphazard distribution of genres in the literature category. There seemed to be one, or often two, drama tossups per round (not necessarily a problem but it did come at the expense of long-form literature), and no poetry in some rounds and 3 poetry tossups in another. I think this seems to be consistent with other distro issues already mentioned (too much baseball, two college basketball tossups in consecutive rounds) that could be alleviated with more careful editing.

[CPiGuy]

This set was badly edited. Some level of difficulty wonkiness is always to be expected, especially with the DII set and the downconversion process. I am also not a good enough player to really comment on what is and is not difficulty-appropriate, so I'll leave that to others. What is not to be expected, however, is a set that contains:

-- straight-up repeats of significant question content
-- other bad feng shui issues, like two tossups on British political parties in the same packet
-- several absurdly sparse answerlines that cause players to get cheated out of points or necessitate contentious protests
-- a bonus answerline where the printed answerline is literally just wrong and has no connection to the rest of the question

etc.

(I'll elaborate on the specific issues in the DII specific discussion thread.)

It was extremely obvious that none of the "set editors" actually read what they purported was the final version of the set at all, because many of these issues were extremely easy to spot and extremely fixable. To be perfectly frank, the only other college-level quizbowl set I've seen that had this many obvious editing issues that tangibly affected playability was the first edition of Math Monstrosity -- a tournament I single-handedly wrote and edited less than six months after first learning how to play quizbowl. NAQT is a professional organization that employs very good and very experienced writers and editors. This is an embarrassing product for such an organization to put out.

[AGoodMan]

Relatively minor, but there was one round with three straight Russia-related tossups: Red Square, Sholokhov, and Michael I.

[Judson Laipply]

I just want to echo Jacob’s sentiment that I am also an NAQT fan and generally like SCT and ICT but this was unacceptably bad.

If this is the last NAQT tournament i play (which is possible given how bad the SOS modifier for D value is) it would really be a (personal) tragedy.

[Auger recombination]

As a reader of this set, something that may not have been evident to the players is that this set contained far more typos than are acceptable. Specifically, it was not uncommon to have to reread a sentence because there was an important word missing that I had to fill in myself. One that I remember particularly well was that the Bartok bonus spelled his name differently in each of the bonus parts.

[naan/steak-holding toll]

I'd also be remiss if I didn't post, yet again, to suggest that NAQT could probably do to have a higher explicit distribution for world literature - maybe 8/8 minimum across a set. Requiring 1/1 per packet in the past hasn't worked that well at regular tournaments, but the current amount seems really low for a subject that a bunch of people engage with - particularly when compared with, say, opera (and many of this set's opera questions felt functionally like literature tossups).

[Zealots of Stockholm]

As a reader of this set, something that may not have been evident to the players is that this set contained far more typos than are acceptable. Specifically, it was not uncommon to have to reread a sentence because there was an important word missing that I had to fill in myself. One that I remember particularly well was that the Bartok bonus spelled his name differently in each of the bonus parts.

I'd like to echo this, and specifically note that I was surprised by it. My past experiences hosting on NAQT sets has led me to expect that they generally have fewer typos and grammatical errors than a typical mACF set (both at the HS and college levels), and this was not the case while reading the set yesterday.

[caroline]

As a reader of this set, something that may not have been evident to the players is that this set contained far more typos than are acceptable. Specifically, it was not uncommon to have to reread a sentence because there was an important word missing that I had to fill in myself. One that I remember particularly well was that the Bartok bonus spelled his name differently in each of the bonus parts.

I'd like to echo this, and specifically note that I was surprised by it. My past experiences hosting on NAQT sets has led me to expect that they generally have fewer typos and grammatical errors than a typical mACF set (both at the HS and college levels), and this was not the case while reading the set yesterday.

As another reader, I'll agree with this as well. I remember at least one bonus part missing a pronoun, at least two bonuses mentioning an answerline which had been repeated earlier in the question (e.g. I think someone brought up the bonus mentioning Shinto temples in the first part and asking for Shintoism in the second), and various other grammatical errors that I found difficult to parse.

[cruzeiro]

As a moderator, I appreciated the intent of the complicated answer line warning, but I found that putting it at the very top of the pack meant that in practice, I completely forgot about where the complicated answer actually was and to pay extra attention to it once I got into reading the pack. So it proved more or less useless to me since it wasn't in front of the complicated question itself.

[Cheynem]

Yeah I might do something like put a WARNING symbol in front of the "complicated answerline" tossups.

[Victor Prieto]

As a moderator, I appreciated the intent of the complicated answer line warning, but I found that putting it at the very top of the pack meant that in practice, I completely forgot about where the complicated answer actually was and to pay extra attention to it once I got into reading the pack. So it proved more or less useless to me since it wasn't in front of the complicated question itself.

I would imagine that this would be particularly helpful for timed rounds, where readers could use the guideline at the beginning of the packet to immediately skip to and read the tricky answerlines during the break between rounds. That way, they don’t spend valuable time reading the answerline while the clock is running.

[VSCOelasticity]

As a moderator, I appreciated the intent of the complicated answer line warning, but I found that putting it at the very top of the pack meant that in practice, I completely forgot about where the complicated answer actually was and to pay extra attention to it once I got into reading the pack. So it proved more or less useless to me since it wasn't in front of the complicated question itself.

I would imagine that this would be particularly helpful for timed rounds, where readers could use the guideline at the beginning of the packet to immediately skip to and read the tricky answerlines during the break between rounds. That way, they don’t spend valuable time reading the answerline while the clock is running.

Yes, and at NAQT tournaments I've staffed they have explicitly instructed moderators to do this.

[Fado Alexandrino]

Is there a way for NAQT to guarantee a distribution within the first 22/22 of a pack? Also, considering those two extra questions don't even constitute the requisite number of tiebreakers, I don't get why packets aren't 25/25 or 22/22 with an extra tiebreaker pack.

[naan/steak-holding toll]

Is there a way for NAQT to guarantee a distribution within the first 22/22 of a pack? Also, considering those two extra questions don't even constitute the requisite number of tiebreakers, I don't get why packets aren't 25/25 or 22/22 with an extra tiebreaker pack.

Personally I'd be fine with 24/24, though I understand not all sites have the reader quality to pull this off and have the tournament run on time.

[jinah]

As a moderator, I appreciated the intent of the complicated answer line warning, but I found that putting it at the very top of the pack meant that in practice, I completely forgot about where the complicated answer actually was and to pay extra attention to it once I got into reading the pack. So it proved more or less useless to me since it wasn't in front of the complicated question itself.

I would imagine that this would be particularly helpful for timed rounds, where readers could use the guideline at the beginning of the packet to immediately skip to and read the tricky answerlines during the break between rounds. That way, they don’t spend valuable time reading the answerline while the clock is running.

I agree with Victor that this makes more sense for timed tournaments, but even for those I think having some kind of warning would be helpful. The problem might be mitigated by having more "Note: Description acceptable" or "Note: author/composer/artist/etc and type of work required"-style notes, which I think might have been helpful for a few questions (the tossups on King Tut's curse and Rembrandt self-portraits come to mind).

Generally, I felt the literature was generally harder to power than the rest of the categories; it may have been the specific teams I saw, but I think I also saw far fewer lit powers than in the rest of the "big three."

[vinteuil]

I agree with Victor that this makes more sense for timed tournaments, but even for those I think having some kind of warning would be helpful. The problem might be mitigated by having more "Note: Description acceptable" or "Note: author/composer/artist/etc and type of work required"-style notes, which I think might have been helpful for a few questions (the tossups on King Tut's curse and Rembrandt self-portraits come to mind).

I believe that there is still no way to implement these within NAQT's system.

[Valefor]

As a reader of this set, something that may not have been evident to the players is that this set contained far more typos than are acceptable. Specifically, it was not uncommon to have to reread a sentence because there was an important word missing that I had to fill in myself. One that I remember particularly well was that the Bartok bonus spelled his name differently in each of the bonus parts.

Was this in DI or DII? I'm looking at these questions and don't see this, but it's possible that someone else has already fixed this.

[DavidB256]

This post isn't about the set, but I'd just like to ask about how there are no statistics whatsoever reported for the NC State mirror of SCT. No team rankings were ever made available (besides first place for either division), no individual scorers were awarded, there is no page for the tournament on the hsquizbowl tournament database, and the NAQT page for the tournament shows results simply as "Not available".

[Important Bird Area]

As a reader of this set, something that may not have been evident to the players is that this set contained far more typos than are acceptable. Specifically, it was not uncommon to have to reread a sentence because there was an important word missing that I had to fill in myself. One that I remember particularly well was that the Bartok bonus spelled his name differently in each of the bonus parts.

Was this in DI or DII? I'm looking at these questions and don't see this, but it's possible that someone else has already fixed this.

I fixed this yesterday evening; Finn's concern was valid for the version of the set as it existed on Saturday.

[setht]

The set editors in DI were Andrew Hart (head editor), Rob Carson, and Billy Busse (science). The set editors in DII were me (head editor), Larissa Kelly, Jeff Hoppes, and Samer Ismail (science). I want to thank my fellow set editors, and the writers and subject editors, for all their work on the sets.

[setht]

Speaking only for myself, I think the obvious takeaway is that I didn't give myself enough time to work on the set. It's embarrassing to have missed silly things like "Béla Bartók/Barók/Bela Bartok" and "I have just replaced this bonus part text with clues for John Dryden, but neglected to update the answer line properly."

Regarding subdistributing: I imagine at least some of this can be automated, but at the moment the things people are complaining about are all things our system does not check for, and which set editors have to look out for and try to fix by hand . . . which requires time. For instance, as Mike said while discussing the sports subdistribution, we have reserved space for basketball but don't have further requirements for college basketball vs. NBA; we have an Any Sports section but no automatic check in place to make sure it isn't overrun by baseball (or jai alai or whatever).

As a separate matter, we also currently don't have automated checks for things like "there's a reasonable amount of organic chemistry in this set, but the o chem tossups are all clumped in the last 6 rounds." A heartfelt thanks to Billy Busse for catching and reporting a couple clumping issues in the assignments of various science subcategories (which were then fixed by hand). Unfortunately we did not catch some non-science clumping issues.

[setht]

I think this set’s failures can be directly attributed to a lack of oversight within NAQT’s editorial system.
.
.
.
There is no particular set of guidelines for what set editors (FEED and SEED) should do beyond ensuring that questions are packetized correctly, factually correct, and typo-free. Nonetheless, it is understood that those entrusted with these roles will shape the set into a consistent whole. I remember the two SCTs that Stephen, Auroni, and I edited. It took a HUGE amount of work to make the middle clues and bonus parts consistent. But there was nobody making sure we would do that; with so many editors involved, I often found that I had to snap myself out of the “somebody else will deal with this, it’s so obvious” mentality. And unlike with other sets, the editors are not necessarily advertised, and take a back seat to “NAQT” when it comes time to talk about the set’s successes and failures.

To clarify a bit, in NAQT's framework there is a clear head editor (which is internally called FEED or First-Eyes EDitor). Andrew Hart was the head editor for the 2017 and 2018 DI SCTs. I don't know what Andrew did or did not say to Jacob/Stephen/Auroni while working on those sets, but my own experience in working with guest editors has been that I was very aware that I was working with excellent, veteran guest editors, and I didn't think it would be appropriate for me to tell them things like "now remember, make sure the middle clues are consistent." Possibly Jacob's saying that we haven't done a good job of communicating that all of the set editors are expected to help with these things in all the questions they look at. If that's been an issue for sets I've worked on, I certainly apologize, and I'm sure Andrew feels similarly.

We do generally avoid tying any specific question's success or failure to a specific person during post-tournament discussion, but we certainly want to recognize and thank our set editors—especially our guest editors!—for their help, which we greatly appreciate. I have to say that I'm surprised and embarrassed to find that it looks like we did not publicly thank the set editors for the 2019 SCT or the 2017 SCT. (And I can't even find the discussion threads for the 2018 SCT—what happened there?)

In the spirit of better late than never, I would like to thank the following SCT set editors:
2019
DI: Andrew Hart, Kyle Haddad-Fonda, Billy Busse
DII: Larissa Kelly, Jason Thompson, Neilesh Vinjamuri

2018
DI: Andrew Hart, Auroni Gupta, Jacob Reed
DII: Larissa Kelly, Matt Weiner

2017
DI: Andrew Hart, Stephen Eltinge, Jacob Reed
DII: Jonah Greenthal, Larissa Kelly

[Borrowing 100,000 Arrows]

Jacob and Will pretty much summed up my feelings about this set. This was probably my least favorite set of the last two years, stocks in power, extreme difficulty swings, etc. I know it's hard to get guest editors for SCT, but it's kind of disappointing for such a flagship tournament to be so mediocre.

[setht]

Speaking only for myself, I think the obvious takeaway is that I didn't give myself enough time to work on the set.

And, to actually address concerns regarding the 2020 ICT: I think by far the most important component of the solution is to have editors spend more time working on the sets. I will do what I can to free up the schedules of the relevant editors, encourage everyone involved to start early, get conversions started early for the folks working on the DII set, etc. I will also try to help make sure everyone's clear on who is doing what, and to detect and fix clumping and subdistribution issues.

[t-bar]

As a separate matter, we also currently don't have automated checks for things like "there's a reasonable amount of organic chemistry in this set, but the o chem tossups are all clumped in the last 6 rounds." A heartfelt thanks to Billy Busse for catching and reporting a couple clumping issues in the assignments of various science subcategories (which were then fixed by hand). Unfortunately we did not catch some non-science clumping issues.

"Clumping," in general, is a difficult problem to solve, especially given the number of fine-grained subcategories in NAQT's distribution. I suspect that this is particularly salient in the sports distribution, where subcategories are especially obvious and people are likely to have preferred small slices of the distro that they want to see. To some degree, I think we should be understanding that a perfect subdistribution is impractical, and not NAQT's first priority.

However, I think that focusing on the distribution over the full 16-packet set is a bit of a red herring. If I'm interpreting NAQT's results correctly, no team played packet 16 of the DI SCT set last year. The required number of games for all teams is 9, and the median and modal number of games played is 10. Even accounting for byes, the median and modal final packet that teams hear is still #10. Only 16 out of 67 DI teams heard a packet after #12. I haven't combed through other years and divisions, but I'd guess they show similar results. Has NAQT considered implementing some minimum quota of each subdistribution that should occur in the portion of the set that the vast majority of teams hear? For example, if a particular subdistribution is allotted 2/2 over the entire set, how hard would it be to ensure that at least 1/1 appears in the first 10 packets? I imagine that asking for exactly half of every subdistro in the first 8 packets is a bit much, but one has a little more wiggle room in 10 packets.

Of course, one could make similar arguments for many high school tournaments run on IS-A and IS packets, so I understand if NAQT feels that this is opening a door to too much micromanagement. However, I think this is a question worth discussing.

[theMoMA]

I thought I'd offer a few reflections now that the discussion has mostly played out.

I've been the primary set editor of SCT for many years. It strikes me that, from an outsider's perspective, it might be difficult to understand the changes to SCT production that have happened in that time. When I first started editing the tournament, it was very difficult to get enough questions to fill the set. I often had to write a large percentage of the questions myself. Although these early sets are (mostly) remembered fondly, I suspect that's more to do with the dramatic break they represented from the previous SCTs than their objective quality, which I'm sure is lower than people would expect today. In future years, we improved the system of bringing on guest editors from active programs, who wrote large percentages of the set (and I contributed large percentages most years as well). In the last two years, we've gone away from that model, and a large percentage of the questions have been written by our high-volume writers and a few recent graduates who had stockpiled up many questions at higher levels that were finally released into the pool of available questions when they left school. I believe I wrote a single question in this year's set, which was the bonus on Emmy Noether.

The pool of subject editors has also changed quite a bit over the years. In the earlier years, the editor pool was fairly broad, and it was often difficult to standardize the various approaches to editing. In the middle years, there was a small group of core NAQT editors who dealt with most of the questions, and most of us had a relatively consistent approach. In more recent years, as NAQT's core members have taken on different duties, the pool of editors has expanded, and achieving consistency in subject editing has been a little harder as a result.

Quite frankly, moving from the "guest editor" model, in which a large quantity of questions were written specifically for the set by its editors, to a "high-volume" model, in which the set is largely generated from a pool of existing questions hanging out in the void, has been challenging to navigate, as has the re-proliferation of higher-difficulty subject editors. It's not that these are untalented writers and editors; they're people who write and edit very well, in fact. But having such a production model makes it more difficult to see how a set is shaping up on a question-to-question basis, and so the old method of simply spending a decent amount of time looking over every question and flagging the ones that need more work can miss some of the bigger-picture items, such as the subdistributional breakdown of the sports category or more nebulous qualities about the "feel" of the questions. I know I look at these last couple SCTs and think that, while I wouldn't go so far as Jacob, they aren't as good as the sets that resulted from the "guest editor" model. Those sets were more artisanally produced and have a more coherent feel, because a lot of the questions were both written and edited to go in that specific set.

And yet, under the pressure of finishing the work, it is difficult to justify kicking a perfectly good question from the set when its only sin is that it sort of "feels" like some of the other questions in the set. And it is difficult to notice the ways that the set is shaping up until you actually read it as played. And sometimes you look at a question, verify that it's factually and grammatically correct, and figure that the clue structure is fine, even though it's not exactly as you might have written it yourself (or rewritten it had you had infinite time), but it turns out that the clues were not ordered optimally, or that the lead-in was more famous than you thought, or the purported easy part is not, in fact, easy enough, or whatever. It's even more difficult to ensure a perfectly even playing experience when many of the categories are "extra-canonical," such as sports or general knowledge, and it's hard to guess just how much the player base actually knows about those subjects, even though they are (to my mind at least) interesting and worth asking about.

There are some additional and largely uninteresting challenges to producing a DI and DII version of SCT and ICT, especially when it comes to being able to know the "final" packet form of the set. Because the DII set includes many questions that the DI set does not, and vice versa, each question movement needs to be harmonized between sets to ensure that it doesn't create unforeseen packetization issues there. This means that, even though I may read through packets as they exist at a given time, it's hard for me to know whether questions might have to move around to accommodate changes in the DII set, which is typically on a later trajectory for completion, simply because many of its questions are converted from DI. The DII editors also have a big challenge of taking conversions, which are often imperfectly generated, and trying to hammer everything out into an even and fair set. Each converted question has undergone many revisions that often rip out large chunks of old text and replace them with large chunks of new text, which can often leave ragged ends that are difficult to notice, no matter how hard you try, such as answer lines that weren't updated or missing or vestigial words here and there. And of course it's usually done under intense time pressure, which is only exacerbated in DII. I don't know if this conversion process can ever be precisely ideal, although I think it's more than good enough for the purpose of picking qualifiers and crowning a champion. All this is to say that there are difficulties inherent in DI/DII set production that, although they can be managed better than they were this year (as Seth says), make the ideal subdistribution and arrangement of questions a little harder than some of the posts above would imply.

I say all this as an explanation of difficulties, not as an excuse for outcomes. I would like for SCT again to receive near-universal praise, and I think that's possible with some of these challenges identified and more squarely in mind during editing. I also agree with Seth that it would be good to expand the amount of time that the editors have to work on the set, and I would go a little further in saying that I would like to start active management of writing and editing these sets far in advance of the deadline, so that it's easier to manage subdistributions and other nebulous items without feeling bad about kicking questions so close to the deadline. I would be open to working with guest editors again as well, although that can have tradeoffs. And of course I will welcome any technological solutions to subdistributional issues and the like.