So, as I mentioned over in the Illinois thread, here's a longer post about NAQT policy and distribution change.Oliver Ellsworth wrote:I'm fully aware that NAQT is simply doing what it does with its screwy distributions and strange answer selections. What I'm lamenting now is the fact that, in a position to touch an enormous number of quiz bowl teams, NAQT continually chooses to ignore the advice and criticism it receives beyond invidual questions or packets. Yes, I like that packets are retroactively edited so they're improved for future use, and I like that NAQT acknowledges when it makes dumb answer choices. But ultimately, NAQT sets still don't have what it takes to be esteemed at the level of HSAPQ and many housewritten sets. The differences are very obvious. Yes, NAQT is huge and it's growing, but as stated before, people often have no choice but to rely on it, so the numbers are a bit inflated. In fact, NAQT's growth is a simply a more urgent sign that things need to change. NAQT could be the single most powerful proponent of the activity it professes to advance, but instead it continues many of its silly ways. Wasted potential just makes me so sad....
Of course NAQT is well aware that many players who post on this forum would like us to change our distribution. If distribution change was just a straightforward matter of improving the quality of our quizbowl, I have no doubt that we would be happy to adopt many of the changes suggested here.
Assume for the moment (an assumption we don't necessarily agree to!) that proposed distribution changes would, in fact, improve the quality of NAQT's sets. Now I'd like to switch gears and talk about the cost of distribution change.
I hope everyone will agree that dead tossups are bad quizbowl. For example, the letter from the circuit asking for NAQT reform wrote: "we are entirely sure that what “average” teams want is easy questions, not bad questions." (Note that Zahed's comment above separates question quality and distribution issues: "I like that NAQT acknowledges when it makes dumb answer choices. But ultimately, NAQT sets still don't have what it takes to be esteemed at the level of HSAPQ and many housewritten sets" so I will assume for the moment that NAQT's questions meet a minimum standard of quality within any given category.) It logically follows that making a set more difficult makes it less attractive to average high school teams. (I'll exclude for the moment extremely easy sets aimed at middle schoolers.)
Now then: would distribution change make our sets more difficult? The answer is yes. Here are our categories, placed in order from "most often answered" to "least often answered." I've marked our own target for conversion rates, and the average conversion rates (including and excluding computational math). Sample size, by the way, is 19,546 IS tossups read between 2004 and 2009.
religious literature
mythology
---------------- (NAQT's stated conversion target)
miscellaneous
current events
general knowledge
geography
popular culture
children's literature
history
theology
literature (popular genres)
--------------------------- (average conversion rate, excluding comp)
sports
--------------------------- (average conversion rate, including comp)
science
literature (excluding myth, religion, kiddie/pop lit)
fine arts
social science
philosophy
computational math
Some notes:
Computational math, as always, is a conversion outlier. High school teams are surprisingly good at myth and religion. Apart from those exceptions, it is generally true that categories with below-average conversion rates are those our critics would like to see increased, and categories with above-average conversion rates are those our critics would like to see reduced:
below-average conversion: (literature, fine arts, social science, philosophy)
above-average conversion: (current events, geography, pop culture, miscellaneous/general knowledge)
Here's an experiment: assume for the moment that we at NAQT decided to switch from the existing IS set distribution and produce IS #94 on the ACF distribution. 31% of the set switches categories. What effect would this have on the difficulty of the set?* An ACF-distributed set would be 1.86% more difficult (3.63% more difficult exclusive of computational math).
I'm sure that's a tradeoff that many teams reading this would make in two seconds flat- sets that are 3-4% more difficult in exchange for improvement in the categories for 31% of the questions. But it's a much more difficult decision to make for a new team that's already struggling with the difficulty of our questions, and consequently for NAQT as a business. To sum up, I would like to suggest that Zahed's post has the causality wrong between our distribution and our market position: we don't believe we're wasting an opportunity to provide new teams with better-distributed questions. We believe that our existing distribution plays a small part in maintaining our outreach to teams that are new to good quizbowl.
*(Notes section:
-This argument assumes that the difficulty of the category remains constant as its share of the distribution expands or contracts; this may be a bad assumption. It might, for instance, be possible to improve our conversion numbers by preferentially selecting the strangest, most difficult tossup answers to be removed. I'm not at all sure we're capable of doing that, given the equally consistent criticism on this forum that our answer selection process is substandard.
-Alternatively, we could increase conversion even as we expand the distribution by repeating the same clues and answers more frequently. Since this is how middle clues become stock, I think this solution is a blind alley that would simply reward teams who study old packets and make quizbowl even more of a self-referential activity than it is now; I don't think it would necessarily help in reaching out to new teams. (Worth noting, by the way, that the general trend is that the harder-to-convert categories -lit, fine arts, phil- are *already* more strongly canonical than the rest of our sets. I'm not sure which direction that runs- whether this is because weak teams haven't learned the canon, or because our subject editors are aware of conversion problems and try to push the answer space toward the best-known subjects. A mix of both wouldn't surprise me.)
-All of the above should be distinguished from prescriptive arguments about the distribution (Of the form: "X% of popular culture questions is too much for academic quizbowl.") Obviously NAQT shares this argument, for certain values of X ("why don't you adopt the TRASHionals distribution, I bet novices would rather play that than actual literature and science?"), but where to place that line is not something we believe is a settled question.)