let's talk about conversion stats

[Important Bird Area]

The following are the conversion rates for 19,563 IS set tossups read between 2005 and 2009.

religious literature 90.4
myth 86.5
--------------------------------------------------(stated target, 85% conversion)
mixed 84.1
current events 82.9
general knowledge 81.5
geography 81.4
pop culture 80.0
children's literature 77.0
history 75.0
theology 74.9
literature (popular genres) 73.8
--------------------------------------------------(average excluding math comp 73.1)
sports 71.7
--------------------------------------------------(overall average 70.8)
science (not math comp) 70.0
literature (not religious, myth, kiddie, or pop lit) 66.0
fine arts 65.5
social science 60.8
philosophy 50.0
math comp 41.5

Edit: fix table

[Important Bird Area]

Even so, with (for example) the current 2/2 econ per packet set, that comes out to about 20/20 econ per year for NAQT. If there aren't even 20 reliably convertible tossups in the high school econ canon, I guess there's no hope for increasing social science's share of the distribution. That's a shame. Out of curiosity, what categories are converted at that 85% target rate?

I still think something needs to be done to reduce the trash. Maybe just increase the big three's share to compensate or something.

Since these questions are across tournaments, depending on the amount of overlap among questions, 20/20 econ in a year is feasible.

Here's some answer space. Your econ tossups and easier bonus parts from the 2008-09 IS sets. (Note that: 1. we only produced nine sets that year and 2. those sets only contained 1/2 or 2/1 econ, not the 2/2 proposed above. So that adds up to 13/14 econ rather than 20/20.)

Arthur Laffer, indifference curve, John Nash, The Wealth of Nations, inflation, savings, Philips curve, JM Keynes, Alan Greenspan, tariffs, Communist manifesto, substitute goods, patents, Thorstein Veblen, macroeconomics, Thomas Malthus, Gresham's Law, monopolies, Karl Marx, progressive taxation, demand curve, David Ricardo, Adam Smith

Note that we had a few repeats as well as things that tend toward the neighboring disciplines like philosophy and history.

[nobthehobbit]

Some initial thoughts:

The higher conversion rate on popular culture over sports suggests to me that (as I've advocated elsewhere) sports should be reduced, perhaps brought to the same weight as the TV, Music and Film parts. (As I noted in the collegiate thread, CULT has equal weights for sports, music, TV, movies, and other.)

Under what top-level distributions do children's literature and literature (popular genres) fall? Where would a question on Dune fall? What about The Time Machine? Oryx and Crake? Redwall? (I'm just trying to get a rough sense of where NAQT draws the lines between various types of lit; I know where I'd put each.)

Math comp should just go (and from HSNCT, too).

I'm honestly not sure if 85% is attainable. In games involving experienced teams, sure, but they can't make up (in this regard) for the novice teams, or the teams seeing pyramidal quizbowl for the first time, that is, the teams unlikely to convert many tossups. (Indeed, some experienced teams don't play much NAQT at all: according to your website, State College played only IS-95 this year, and Maggie Walker played only IS-90 and IS-95. Of course, that's not exactly a large sample size.) It would be interesting to note which schools send teams (and which teams they send) to IS tournaments held against ones like EFT and ACF Fall, to which many high school teams now go, or to HSAPQ/independent tournaments held against IS ones. I think the idea of an IS-C set was floated before (harder than IS, easier than SCT, but hopefully still suitable for collegiate teams), as was the idea of an NAQT set with mACF distribution. It would be interesting to see how those go over, though, as I understand it, NAQT's just about at the limit of its writing capacity (so such sets would necessitate a decrease in IS/IS-A sets?).

[Important Bird Area]

Under what top-level distributions do children's literature and literature (popular genres) fall? Where would a question on Dune fall? What about The Time Machine? Oryx and Crake? Redwall? (I'm just trying to get a rough sense of where NAQT draws the lines between various types of lit; I know where I'd put each.)

The top-level distribution is literature. All of those answers would fall under literature (popular genres), with the possible exception of Oryx and Crake, which hasn't yet appeared as an answer in its own right (just a middle clue for Atwood questions).

[Important Bird Area]

I'm honestly not sure if 85% is attainable. In games involving experienced teams, sure, but they can't make up (in this regard) for the novice teams, or the teams seeing pyramidal quizbowl for the first time, that is, the teams unlikely to convert many tossups. (Indeed, some experienced teams don't play much NAQT at all: according to your website, State College played only IS-95 this year, and Maggie Walker played only IS-90 and IS-95. Of course, that's not exactly a large sample size.) It would be interesting to note which schools send teams (and which teams they send) to IS tournaments held against ones like EFT and ACF Fall, to which many high school teams now go, or to HSAPQ/independent tournaments held against IS ones. I think the idea of an IS-C set was floated before (harder than IS, easier than SCT, but hopefully still suitable for collegiate teams), as was the idea of an NAQT set with mACF distribution. It would be interesting to see how those go over, though, as I understand it, NAQT's just about at the limit of its writing capacity (so such sets would necessitate a decrease in IS/IS-A sets?).

It is an interesting hypothesis that there might be a feedback loop at work here (the best teams play fewer IS sets, leading to lower conversion rates for fine arts, leading NAQT to resist increasing that category, repeat). (But note that most of the data I posted is several years old, before HSAPQ existed and when many fewer hs teams played college tournaments).

As for new sets:

I consider a higher-difficulty set in the near future to be unlikely, both because tournaments like ACF Fall and EFT do a good job of serving that audience, and because the additional question production would distract from DII SCT and HSNCT that are of broadly similar difficulty (harder than regular-season high school, easier than DI SCT).

Now, redistributing sets would be much easier to work with (because we would only have to generate a comparative handful of questions to bring a given IS set to, say, the SCT distribution). It would certainly be interesting to see whether an IS set using our college distribution (or indeed an ACF distribution) would outsell our existing high school distribution.

[nobthehobbit]

It is an interesting hypothesis that there might be a feedback loop at work here (the best teams play fewer IS sets, leading to lower conversion rates for fine arts, leading NAQT to resist increasing that category, repeat). (But note that most of the data I posted is several years old, before HSAPQ existed and when many fewer hs teams played college tournaments).

Well, even the best teams will play one or two IS sets a year, if only to guarantee HSNCT bids (or claim a "state champion" title that'll look good to school admin, or something). But yes, I wouldn't be surprised if such a feedback loop existed, and it almost certainly affects categories like social science, too.

As for new sets:

I consider a higher-difficulty set in the near future to be unlikely, both because tournaments like ACF Fall and EFT do a good job of serving that audience, and because the additional question production would distract from DII SCT and HSNCT that are of broadly similar difficulty (harder than regular-season high school, easier than DI SCT).

Now, redistributing sets would be much easier to work with (because we would only have to generate a comparative handful of questions to bring a given IS set to, say, the SCT distribution). It would certainly be interesting to see whether an IS set using our college distribution (or indeed an ACF distribution) would outsell our existing high school distribution.

HFT would be another such higher-difficulty set (as I understand it), and also serves as an NSC qualifier (unlike ACF Fall or EFT).

As for the idea of changing the distribution of an IS set to that of SCT (or an mACF one), I would guess that it would work the first time (for the novelty if nothing else), but whether it worked after that would be dependent on how much "crazy NAQT stuff" was still in there. (Technology, I'm looking at you, and I note you wouldn't be much different.)

But there is still the difficulty thing: if teams play the tournaments they think will help them improve most, the top teams (ie the ones that would increase your conversion rates) will prefer other sets to IS sets. (Extreme case: 5 of the 8 players in the HSNCT final played ACF Nationals, and only one was under 20 PPG; SC put up 12.3 PPB on a set on which Yaphe's Stanford got 18.5; I wouldn't have been surprised if, had MW played, SC and MW would have been in the D2 final.) It would be interesting if we had this sort of conversion data for HSAPQ sets, or HFT IV+mirrors. Changing the distribution of a set might shift the answer space to where some teams will find it more to their liking, but they may still find it too easy. It might work as some sort of coordinated special event, not conflicting with other major (HS or collegiate) tournaments.

[Down and out in Quintana Roo]

Are we going to see this sort of conversion data for HSNCT, like you did last year? That was a really cool and helpful thread, when you told us about the least and most converted tossups, etc.

[Stained Diviner]

It seems like NAQT has collected a lot of good data over the years, and I don't know whether they are making good use of it. (Honestly--I don't know.)

Here might be one idea you could try:
1. Give each subject editor a list of past tossups and conversion rates.
2. Based on those conversion rates and the editor's opinion of the merit of a topic, have them come up with a maximum number of times that the topic should appear during the year in IS/A Sets. For some answers, that maximum number might be zero, which would be included on the list, though it would not preclude writers from using that topic as a clue or bonus part. There would also be a number of topics on which there is not enough data on, meaning that it wouldn't be listed but writers would have the option of writing on it.
3. NAQT would make this information available to its writers and give them ways to claim one use of the answers they wanted to use. The editors would also check submissions to make sure they were claimed.

Though I am suggesting this as a limit, I hope the impact it would have would be to encourage certain answers. Basically, if I wanted to write some Econ questions, I would check the list and see that the editor has determined that questions on, say, Adam Smith, are good and that there have not been any such questions written yet for this year, so I would write that question instead of one on Gresham's Law or whatever topic the editor is trying to discourage repeats of.

Of course, this demands some extra lead work by the editors, and I'm not in a position to determine whether it would be worth it. It might be useful at first to just try this out on fine arts, social science, and philosophy, where the answer space is fairly small and the current conversion rate is far from the target.

[vcuEvan]

Where is the conversion data collected from? As in, which tournaments are included?

[Important Bird Area]

Are we going to see this sort of conversion data for HSNCT, like you did last year? That was a really cool and helpful thread, when you told us about the least and most converted tossups, etc.

Yes, there will be an HSNCT conversion discussion. We have to finish tallying the scoresheets first, though.

[Important Bird Area]

Where is the conversion data collected from? As in, which tournaments are included?

The set above is disproportionately Minnesota (R.) and Ohio (Greg Bossick) tournaments, but anyone who cares to send us a box of scoresheets is welcome to do so.

[BRizzle]

Why is NAQT's conversion goal 85%? Is there a reason this number was chosen?
Secondly, are teams really not going to come back to a tournament if NAQT adds more social science and fine arts while cutting down on trash?

[Important Bird Area]

Why is NAQT's conversion goal 85%? Is there a reason this number was chosen?
Secondly, are teams really not going to come back to a tournament if NAQT adds more social science and fine arts while cutting down on trash?

1. To my knowledge, 85% is a relatively arbitrary target, but one chosen with the twin goals that: A. most questions should be answered by someone but B. the questions should still be challenging.

2. Yes, some of them will (not necessarily because they dislike those particular categories, but because dead tossups are the enemy of circuit expansion).

[MahoningQuizBowler]

Where is the conversion data collected from? As in, which tournaments are included?

The set above is disproportionately Minnesota (R.) and Ohio (Greg Bossick) tournaments, but anyone who cares to send us a box of scoresheets is welcome to do so.

Along these lines, I have all of the scoresheets (as well as the 870 completed surveys) from HSNCT. I am tallying the survey results now and anticipate being done with those by the end of the weekend. From there, I'll move on to the conversion stats from the scoresheets -- both playoff and prelim, tossup and bonus.

If anyone has scoresheets from other events to send, please message me and I will send you my address.

[cvdwightw]

Secondly, are teams really not going to come back to a tournament if NAQT adds more social science and fine arts while cutting down on trash?

Well, I'm going to divide NAQT customers into five groups, based on their likely response to this:

1. "We like HSAPQ (or other ACF-like tournaments) and either prefer it to NAQT or have no preference regarding the two, so we have no problem with NAQT adopting a distribution more similar to HSAPQ."

2. "We play NAQT because we don't know of the existence of other formats of [good] quizbowl and will continue to play NAQT regardless of distribution."

3. "We like quizbowl and quizbowl-like activities and will play tournaments regardless of quality/distribution."

4. "We prefer NAQT to HSAPQ due to the shorter tossups. We have no problem with an increase in fine arts/social science and a decrease in trash."

5. "We prefer NAQT to HSAPQ because of the prevalence of trash and other non-academic fields. We are strongly against changing the distribution to increase social science and fine arts and decrease trash."

Of these schools of thought, groups 2-4 really couldn't care less about the distribution either because they don't know, don't care, or want NAQT to keep something that has nothing to do with distribution. It really boils down to whether NAQT wants to side with group 1 (the vast majority of schools that post on this board; the vast majority of HSNCT playoff teams) or group 5 (likely a small minority of idiot coaches who think that "Which little piggy had roast beef?" is a fair test of academic material learned by high schoolers through classes and independent study). [EDIT: I guess there's a sixth group that Jeff's talked about, which is people who don't object on principle to less trash until they see that no one can answer the non-trash questions]

Disregarding accusations of "push polling" and questions on things that have little to no relevance in the modern NAQT game (e.g. "Should we re-include wordplay tossups?"), the NAQT surveys I am familiar with have had largely two flaws: (1) some questions have been unclear or failed to provide a response choice that accurately fits respondents' views; and (2) there is a large response bias because they will be filled out and returned by people who care strongly one way or the other (not to mention that I think the only true surveys NAQT does anymore are ICT and HSNCT, which likely have different views than their overall customer base). Accordingly, the results of these surveys are difficult to interpret (I am assuming that the chance of outright abuse, e.g. by a group of anti-mathcomp people all claiming to be coaches from Illinois, are negligible).

I'm not sure whether NAQT does more "private" surveys that they send out to a representative sample of their customer base (if they do, they don't make the results public, at least). I'm not sure whether, if they do, they make any attempt to stratify the results based on number of tournaments attended or HSNCT participation. I am fairly sure that NAQT has a large database of contacts for just about every team that participated in an NAQT tournament, and would therefore have the ability to conduct such a survey. There would still be a response bias, but I think that most coaches would take a couple of minutes to respond to a simple set of questions (The simple 5-point scale "Way too little/Too little/Just the right amount/Too much/Way too much" for the top-level distribution that NAQT has done on previous surveys would suffice), and looking at the results in a variety of different ways could allow NAQT to stop lumping a bunch of disparate schools together into the "silent majority."

[Bananaquit]

I don't really know how indicative this is of the rest of the country, but many lower-level teams in the Mid-Atlantic and specifically in the Richmond area come to whatever tournaments are available regardless of distribution or even difficulty. For example, the Mid-Atlantic Regional Championship at VCU (DII Sectionals) had a couple such teams, despite being hard, and GSAC and HSAPQ tournaments tend to pull just as many local teams as NAQT tournaments in the area do. Because none of these teams tend to convert arts/SS questions (or other hard questions) at a high rate but still play, I would say they probably fall under Dwight's third category. If I'm correct, such teams probably wouldn't mind some distributional changes, since they already do not expect to convert all the questions in the arts/SS/lit categories but still play.

[djones]

While there may be a handful of teams who would not come if trash / non-academic material was reduced or eliminated, I don't think it would be an "irreplaceable" group. For instance, I would venture to guess that 90-95% of the teams in the Southwest Ohio region that I am familiar with have no clue what NAQT, HSAPQ, or any other acronym you want to throw out there means in terms of distribution. Most only know that these tournaments are different from OAC, but don't really know anything else about them.

Of the teams in our region that do know the difference between these acronyms, most (if not all) would attend ANY toss-up tournament held in the area in lieu of attending an OAC event.

In promoting our annual Rowdy Raider NAQT tournament, we draw 20 or so "good" teams from around Ohio, Kentucky and other neighboring states, and 10-12 teams local from our region. I usually have to explain to the local teams the rules for toss-up/bonus as they have been weened on OAC for all of their quiz bowl lives. If the TU/B questions they play at an event have no trash one year after having a 1/1 distribution of it the previous year, I don't think they would even notice, having so little exposure to it anyway. Similarly, if we were to switch from NAQT to HSAPQ questions or a mirrored set, I don't think it would make any difference at all to those fringe teams because they wouldn't know what they are. Of the 10-12 local teams that attend annually, we usually retain 7-8 the next year, and have a few drop off. However, we have never had any problem recruiting some new local teams to take their places.

I know that Ohio is not indicative of the rest of the country just because of the high prevalance of the OAC format in our state, and thus the relative lack of exposure of many teams across the state to the TU/B format (something that many in the Ohio quizbowl community are working hard to change). However, I don't think a distribution change ny NAQT would have any impact on my ability to draw teams to our tournament. NE Ohio may be different given the considerably higher number of events held in that part of the state, but SW Ohio would see little to no impact.

[Excelsior (smack)]

While there may be a handful of teams who would not come if trash / non-academic material was reduced or eliminated, I don't think it would be an "irreplaceable" group. For instance, I would venture to guess that 90-95% of the teams in the Southwest Ohio region that I am familiar with have no clue what NAQT, HSAPQ, or any other acronym you want to throw out there means in terms of distribution. Most only know that these tournaments are different from OAC, but don't really know anything else about them.

This is true. I don't know how many regions are like SW Ohio in this respect - with so little exposure to good quizbowl that all of it seems like some sort of alien entity - but most of these teams probably fall into #3 on Dwight's classification - they play quizbowl because it is an activity that has buzzers. These teams would probably not object to (or even notice) a shuffling around of the academic categories - less geo for more socsci/geo/fine arts/whatever. However,

If the TU/B questions they play at an event have no trash one year after having a 1/1 distribution of it the previous year, I don't think they would even notice, having so little exposure to it anyway.

This I disagree with. I presume that most lousy local formats like OAC are generally heavy on the trash, and trash is a staple of the quizbowling experience for teams that are nigh-exclusive consumers of those formats. As you may recall from my commentary on the SW OAC regional this year, teams definitely did notice the sudden disappearance of trash. Because trash is, in a sense, objectively different from all the other academic categories, teams do tend to notice when the amount of trash in a format decreases.

[swwFCqb]

As you may recall from my commentary on the SW OAC regional this year, teams definitely did notice the sudden disappearance of trash. Because trash is, in a sense, objectively different from all the other academic categories, teams do tend to notice when the amount of trash in a format decreases.

I have a feeling the notice of the absence of trash (in this instance) was due more to it being an OAC regionals set as opposed a regular season TU/B set. The thing you have to remember is that OAC regionals is the pinnacle of the year for a majority of Ohio teams, who pretty much just play in their hometown leagues (some of which are in even worse formats than OAC) and only go to a big tournament if it's within a 15 minute drive. Making it to regionals is a huge deal for these teams (many of whom will publicize it to their school boards, local newspapers, and the like), and when you go from one liners that include Black Sabbath in the Fine Arts category and questions on topics like Dora the Explorer in the regionals/state set to rigidly academic questions 3+ lines in length, then these teams will most definately notice. I tend to side with David on this one...these teams are most likely not going to notice 1/1 less trash in a regular season TU/B tournament because they just don't care enough about the format to have any complaints about minor distributional differences (length and difficulty is much more of an issue for these teams).