National tournaments and qualifiers: the same distribution?

Dormant threads from the high school sections are preserved here.
Locked
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 5607
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

National tournaments and qualifiers: the same distribution?

Post by Important Bird Area »

This is your thread to discuss: should national tournaments (such as HSNCT) use the same distribution as their qualifying events (such as IS sets)?

I don't think there's presently a consensus on this issue (either within the quizbowl community or within NAQT).

One argument is that put forward by Charlie in the other thread:
Terrible Shorts Depot wrote:It strikes me as questionable, if not outright unfair, to play Nationals on a different distribution from its qualifying and affiliated tournaments.
The argument being that the qualifying system demands equality of distribution between the sets; otherwise nationals could invite the "wrong" bubble team.

Alternatively: one could argue that the greater strength of the nationals field requires a different distribution.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred

User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 5607
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: National tournaments and qualifiers: the same distribution?

Post by Important Bird Area »

Not speaking for NAQT: I'm quite convinced that the future of quizbowl is the second option, national tournaments that have a different distribution from their qualifiers.

Why? The distribution of regular-season high school sets (and even more so novice high school sets) is going to be limited by what players can answer. We've seen in the other thread that lots of players posting here would like to see IS sets with more fine arts, literature, and social science, even though these categories have poor conversion numbers at their existing lower levels. That leaves NAQT with three choices:

1. We could change the distribution and (say) triple the amount of social science. This will make our regular-season sets harder; it will make quizbowl less fun for newer teams to play and hurt circuit expansion.

2. We could keep the distribution nearly-identical between regular-season sets and nationals. This is fine for regular-season play, but over the long term I'm convinced that it will frustrate the best teams in the country (as they are stuck with a limited canon that doesn't let them display the true depth of their knowledge). It's possible that in the long run this could hurt the legitimacy of HSNCT as a national championship (if enough teams decided to leave over distribution issues in favor of alternative tournaments).

3. We could decide to adjust the distribution between the tournaments. The cost here is loss of fidelity in qualifying events: it's possible that team # 170-something actually does worse on HSNCT questions than the team they beat at their local qualifier (say the next-best team in the county is better at fine arts and worse at geography, and comes one tossup short of qualifying).


None of these options are great, but I'm convinced that (3) is the least harmful. If we have to choose between appealing to teams that marginally qualify for nationals and appealing to the top playoff teams, the right choice is obvious.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5741
Joined: Mon Oct 23, 2006 2:00 am

Re: National tournaments and qualifiers: the same distribution?

Post by theMoMA »

From a pure qualification standpoint, the question for me is whether having a different distro means that the "wrong" teams will qualify somehow. At the upper end, I don't think that this would happen; the teams that benefit from a more rigorous distribution are the ones who are going to qualify regardless of whether there is 1.5% social science or 5% social science. Of course, qualification questions aren't the reasons that these teams want a different distribution, but I agree that the purpose of the regular season is also to interest new teams and allow decent teams to score plenty of points. So on that front, I don't think there is anything practically unfair about having different IS and HSNCT distributions; the teams that matter will qualify.

The other question is whether having a different distro in IS sets benefits certain teams towards the bottom of the qualification heap. If, like Jeff says, the main difference is to minimize the traditionally difficult-to-fill categories at the regular-season level (like social science), I think that the "right" teams will still qualify. The questions will test more of what the teams at lower end of qualification know, and the games should actually be better reflections of the skill difference between these teams.

However, this means that it is extremely important that the categories replacing these difficult-to-fill regular-season categories are similarly academic. If you replace 15% of the HSNCT distribution that's too hard to write at the regular level with trash (or any category that is not very academic), you're doing a disservice to all the teams at the bottom of the pile who qualify for HSNCT on the strength of their prowess in categories that will disappear at the HSNCT itself.
Andrew Hart
Minnesota alum

User avatar
nobthehobbit
Rikku
Posts: 293
Joined: Sun Feb 19, 2006 1:18 am

Re: National tournaments and qualifiers: the same distribution?

Post by nobthehobbit »

Option #3 is best (even if only in the sense of "least of all evils") IMO:

Qualifiers and nationals have different purposes. Qualifiers pick out the teams that are "good enough" to be given a shot at nationals. Nationals pick out the best team. (This applies equally to IS/HSNCT and SCT/ICT, I think.) As Andrew says, the top teams will qualify anyway (well, unless you really skew the distribution terribly), and those same teams are the ones battling for the title at nationals. It's not that important (to me) whether nationals picks out the 107th-best team correctly; it's far more important that it pick out the top places fairly.

So I don't think it's bad to throw lower-ranking teams a bone at qualifying tournaments (IS, SCT). At Nationals, though, and especially tiebreaker (at ICT) and playoff (especially at HSNCT) rounds, the focus should be on which team has the most academic knowledge, and trash doesn't enter into that.
Daniel Pareja, Waterloo, Canadian quizbowl iconoclast

Stats zombie.
William Lyon Mackenzie King wrote:There are few men in this Parliament for whom I have greater respect than the leader of the Co-operative Commonwealth Federation. I admire him in my heart, because time and again he has had the courage to say what lays on his conscience, regardless of what the world might think of him. A man of that calibre is an ornament to any Parliament.

User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: National tournaments and qualifiers: the same distribution?

Post by Mechanical Beasts »

theMoMA wrote:However, this means that it is extremely important that the categories replacing these difficult-to-fill regular-season categories are similarly academic. If you replace 15% of the HSNCT distribution that's too hard to write at the regular level with trash (or any category that is not very academic), you're doing a disservice to all the teams at the bottom of the pile who qualify for HSNCT on the strength of their prowess in categories that will disappear at the HSNCT itself.
That "bottom of the heap" is still the top 15% of the qualification field; they're not the ones we really need to milk tossup conversion out of. The danger that teams 15% and 16% are interchanged by vastly different trash acumen is limited. I agree that you'd be just as well off just pumping the IS distribution full of some other well-converted category.
Andrew Watkins

User avatar
Stained Diviner
Auron
Posts: 4728
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland
Contact:

Re: National tournaments and qualifiers: the same distribution?

Post by Stained Diviner »

I agree that Option #3 is good within boundaries. I'm uncomfortable with something being 1.5% during the season and 5% at HSNCT because teams that play primarily IS during the year are going to get the sense that the category is unimportant and are not going to get significant experience answering questions in it. My sense is that doubling a category is probably an upper limit on how much it should change between IS and HSNCT, though I wouldn't put a limit on how much a category could be decreased.
David Reinstein
PACE VP of Outreach, Head Writer and Editor for Scobol Solo and Masonics (Illinois), TD for New Trier Scobol Solo and New Trier Varsity, Writer for NAQT (2011-2017), IHSSBCA Board Member, IHSSBCA Chair (2004-2014), PACE Member, PACE President (2016-2018), New Trier Coach (1994-2011)

User avatar
nobthehobbit
Rikku
Posts: 293
Joined: Sun Feb 19, 2006 1:18 am

Re: National tournaments and qualifiers: the same distribution?

Post by nobthehobbit »

Westwon wrote:I agree that Option #3 is good within boundaries. I'm uncomfortable with something being 1.5% during the season and 5% at HSNCT because teams that play primarily IS during the year are going to get the sense that the category is unimportant and are not going to get significant experience answering questions in it. My sense is that doubling a category is probably an upper limit on how much it should change between IS and HSNCT, though I wouldn't put a limit on how much a category could be decreased.
Well, this is why distributions are published, and a reason for we as those who know these things to let them be known to those who may not be as much in the know. I'd send the relevant distributions to teams participating in my tournaments, and especially to those who qualify for nationals.
Daniel Pareja, Waterloo, Canadian quizbowl iconoclast

Stats zombie.
William Lyon Mackenzie King wrote:There are few men in this Parliament for whom I have greater respect than the leader of the Co-operative Commonwealth Federation. I admire him in my heart, because time and again he has had the courage to say what lays on his conscience, regardless of what the world might think of him. A man of that calibre is an ornament to any Parliament.

User avatar
cvdwightw
Auron
Posts: 3446
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: National tournaments and qualifiers: the same distribution?

Post by cvdwightw »

theMoMA wrote:If you replace 15% of the HSNCT distribution that's too hard to write at the regular level with trash (or any category that is not very academic), you're doing a disservice to all the teams at the bottom of the pile who qualify for HSNCT on the strength of their prowess in categories that will disappear at the HSNCT itself.
This already happens to some extent with the option of math calculation tossups in IS tournaments and their disuse as HSNCT. Mathcalc is by and large the worst-converted tossup topic and is replaced at HSNCT by roughly proportional amounts of every other category.

My experience is that it is quite rare for a "top 15%" team and a "top 16%" team to consistently be differentiated. Due to the vagaries of packets within sets and catching better teams playing poor/worse teams playing well, teams N and N+1 are rarely the same at two qualifying tournaments. Jeff's problem only exists for borderline teams that play one qualifying tournament; even then, the non-qualifying team would have good reason to apply for a wild card bid. There are a variety of reasons that a consistently-better team on IS questions performs worse on HSNCT questions that have nothing to do with differing distributions; perhaps a team's best player is not playing well that tournament, or draws a harder set of opponents via the card system, or the team's knowledge does not scale well with difficulty. Changing the distribution would not significantly affect a below .500 team's placement more than these already-extant factors already do. In short, I don't think that the reasons Jeff has put out for not implementing option 3 will affect more than a handful of teams that catch a bad break on the one NAQT tournament they play all year.

The IS distribution can be changed to guarantee 4/4 "academic" literature, 2/2 arts, and 1/1 social science per 24/24 by removing a total of 17/18 (just over 1/1 per packet!) from the traditional non-academic (PC, Mixed_Impure) and "overrepresented" (CE, Geography) categories and redistributing it. This means reducing the conversion rate of ~5% of the tossups by 15-20%. Maybe I'm doing horribly bad math but this translates to something like a 1% decrease in overall conversion rate. The only real conversion-based argument NAQT has for not doing this is that it's consistently exhausting the accessible arts/SS space and would thus see even lower conversion rates on those questions than it's already seeing. Furthermore, given the typically well-founded suspicion that the teams that are consistently not converting the arts/literature/SS questions are in general the teams that are not qualifying for HSNCT, there is no conversion-based reason at all to not do the equivalent of this at the HSNCT level.

I've run the numbers estimating the following: 5 academic literature tossups, 7 arts, and 5 SS added; 3 PC, 3 sports, 6 CE, 5 Geo deleted; all 17 added tossups have a 0% conversion rate and the other 343 a conversion rate equal to what Jeff posted in the other thread for their subject. The result is that the overall conversion rate would average 68.3% and the overall non-comp rate would average 69.4%. That's a 2.5% decrease in overall conversion rate and a 3.7% decrease in non-comp conversion. For an average 12-team tournament on 20/20 packets, that's a decrease of between 33 and 49 tossups answered over the entire tournament if NAQT were to replace 17 tossups in non-academic or overrepresented categories with completely unanswerable tossups in underrepresented academic categories. (At a 50% conversion rate in the added tossups, it's a 0.1% decrease in overall rate and a 1.2% decrease in non-comp - between 1 and 16 tossups over the whole 12-team tournament.) This decrease does not appear to provide evidence that tweaking the distribution to provide 4/4 lit, 2/2 arts, and 1/1 SS per 24/24 would significantly affect overall conversion rates. I would highly encourage NAQT to run its own numbers regarding this, especially if/when it has HSNCT conversion data that Jeff has not made public, and come up with a compromise solution that increases the academic content of the game while not giving up much in the way of conversion.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5741
Joined: Mon Oct 23, 2006 2:00 am

Re: National tournaments and qualifiers: the same distribution?

Post by theMoMA »

Two things about HSNCT qualification that really haven't been addressed in this thread:

1) Teams usually have many chances to qualify. It's not really important to distinguish between the top 15% and top 16% teams, since both will probably qualify at some point during the year if they're diligent enough. If a team keeps finishing one out in a strong region, they'll probably get a wildcard spot.

2) Qualification varies by region. The bottom-end qualifiers at your average DC-circuit tournament (and sometimes in other strong circuits like CA, IL, MI, MN, etc.) are potential HSNCT playoff teams. The bottom-end qualifiers in weaker circuits are the most likely to be "wrong" qualifiers on the strength of the non-academic material in the set. There are also fewer tournaments in these regions, so there are fewer chances to get these qualifiers right. You could also argue that sending the wrong teams from these circuits to HSNCT could discourage newer players, though my experience with teams that are overmatched at HSNCT has usually been that they are more excited to be at nationals than they are discouraged that they're not doing well.
Andrew Hart
Minnesota alum

User avatar
Carambola!
Lulu
Posts: 71
Joined: Wed Feb 04, 2009 10:09 pm
Contact:

Re: National tournaments and qualifiers: the same distribution?

Post by Carambola! »

The substitutions for trash that I think are feasible are general knowledge or perhaps current events.
Jeffrey Li
Mission San Jose High '11
Berkeley '15

But I shall forgive you. Some things are beautiful because they cannot be obtained.

Locked