A Modest Proposal Regarding D2 ICT Qualification

Old college threads.
Locked
User avatar
Bosa of York
Rikku
Posts: 362
Joined: Sun Jun 15, 2014 9:16 am

A Modest Proposal Regarding D2 ICT Qualification

Post by Bosa of York »

This year's Rocky Mountain SCT site was won by BYU, a D2-eligible team. However, they did not receive a Tier-5 ICT invitation (for SCT site champions) because fewer than four teams in the combined field were D2-eligible--the qualification rules say that "In the event of a combined-field SCT, there must be four teams from three schools in the team’s division." I think this is a mistake. BYU would have qualified if Colorado A had been made up entirely of new players instead of D2-ineligible ones, or if Colorado B or Colorado Springs had split their rosters into two teams. It would have been easier for them to win the tournament in both of these cases, and it doesn't make sense to me why they should only be rewarded with a bid in the easier of the two scenarios. It seems clear, to me at least, that beating D2-ineligible teams is nearly always harder than beating eligible ones, and I don't see why the aptitude demonstrated by the former should be disregarded when offering bids.

In short, I would suggest that in the future if a D2-eligible team wins a combined-field SCT with four or more teams from three or more schools in total, it should be able to recieve a Tier-5 invitation regardless of the number of teams that are D2-eligible. This precise situation is quite rare, and of course this argument does not apply to D1 teams with too few D1 opponents, or to sites with fewer than four teams. I'd appreciate hearing other people's thoughts on this matter, and in particular critiques of my argument.
Eric Wolfsberg
Bethlehem Central High School 2016
University of Delaware 2020
Stanford 2025 or whatever
Wartortullian
Rikku
Posts: 376
Joined: Mon Dec 19, 2016 10:02 pm
Location: New Haven, CT
Contact:

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by Wartortullian »

As the director of this tournament, I completely agree.

I should also clarify that Kansas State was supposed to attend, but they blew a tire and were unable to make it to Boulder. Had this not occurred, the field would have had enough teams to act as a qualifier.
Matt
Edward Lansdale
Lulu
Posts: 69
Joined: Fri Sep 11, 2009 4:43 pm

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by Edward Lansdale »

Agree. I remember looking at SCT results and feeling bad for BYU; it's not their fault that the Mountain West is a quizbowl desert. Alabama qualified for D1 despite finishing 45th on D-value, so BYU's 46th is scarcely less deserving.
Mirza Ahmed
New York University '12
Keck Graduate Institute '15
jonah
Auron
Posts: 2383
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by jonah »

Over the summer we will review the many changes we made to SCT and ICT this year, including the new qualification scheme, and we will consider changes (including this one) for 2019 and beyond. Thanks for the suggestion!
Jonah Greenthal
National Academic Quiz Tournaments
User avatar
Bosa of York
Rikku
Posts: 362
Joined: Sun Jun 15, 2014 9:16 am

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by Bosa of York »

I figure it behooves me to necro this thread since it seems likely that this situation will happen again. The combined-field Northwest SCT site was won by Washington B, a DII team, but since the only other DII teams at that site were all from Boise State there were only DII teams from two schools, the site did not have the DII teams from three different schools required for a Tier 5 bid, even though the tournament as a whole also had a DI team from Oregon. Washington B is (at time of writing) nineteenth in DII D-value, meaning that as statistics from more sites are added they will almost certainly be pushed out of the top 20 for a Tier 1 bid, and very possibly out of range for whatever Tier 7 bids are granted.

I recognize that there was other stuff going on in the summer of 2018 and that NAQT may have forgotten about this suggestion, or that they may have decided against its merits, but if any group of people with the ability to change the ICT qualification rules thinks that this is a good idea, it may be beneficial to enact such a change before ICT bids are granted if at all possible.
Eric Wolfsberg
Bethlehem Central High School 2016
University of Delaware 2020
Stanford 2025 or whatever
User avatar
Smuttynose Island
Forums Staff: Moderator
Posts: 614
Joined: Wed Oct 21, 2009 9:07 pm

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by Smuttynose Island »

Milhouse wrote: Sun Feb 09, 2020 8:06 pm I figure it behooves me to necro this thread since it seems likely that this situation will happen again. The combined-field Northwest SCT site was won by Washington B, a DII team, but since the only other DII teams at that site were all from Boise State there were only DII teams from two schools, the site did not have the DII teams from three different schools required for a Tier 5 bid, even though the tournament as a whole also had a DI team from Oregon. Washington B is (at time of writing) nineteenth in DII D-value, meaning that as statistics from more sites are added they will almost certainly be pushed out of the top 20 for a Tier 1 bid, and very possibly out of range for whatever Tier 7 bids are granted.
Mike Bentley and I were discussing this on the car ride to SCT and we were both very surprised to learn that NAQT's official qualification policy would not automatically grant UW B a DII bid if they won the Pacific Northwest SCT. Is there some reason why NAQT has not changed this policy?
Daniel Hothem
TJHSST '11 | UVA '15 | Oregon '??
"You are the stuff of legends" - Chris Manners
https://sites.google.com/site/academicc ... ubuva/home
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by Mike Bentley »

FWIW the team received a host bid.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
User avatar
Bosa of York
Rikku
Posts: 362
Joined: Sun Jun 15, 2014 9:16 am

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by Bosa of York »

Is Washington taking their host bid in DII (which is the situation that would obviate this)? I assumed that they'd take it in DI since they have DII-ineligible players. In any event that doesn't change the central issue here, which is something that's going to keep happening as long as there are these really small sites.
Eric Wolfsberg
Bethlehem Central High School 2016
University of Delaware 2020
Stanford 2025 or whatever
Wartortullian
Rikku
Posts: 376
Joined: Mon Dec 19, 2016 10:02 pm
Location: New Haven, CT
Contact:

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by Wartortullian »

Chiming in to say that this proposal is good. It was moot for our site this year, since BYU retained one of their DI players and would not have qualified anyway, but the same thing could very well have happened again.
Matt
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by Mike Bentley »

Milhouse wrote: Sun Feb 09, 2020 8:39 pm Is Washington taking their host bid in DII (which is the situation that would obviate this)? I assumed that they'd take it in DI since they have DII-ineligible players. In any event that doesn't change the central issue here, which is something that's going to keep happening as long as there are these really small sites.
DII. But yes, this year at least, I think the D-Value is clearly underanking Washington B. There's only about 30 points difference between them and Illinois C but they have 5 PPB higher than Illinois C. PPB isn't the only factor in how good a team is, but I don't think it's very controversial to say that 5 PPB is a pretty huge gap between team skill levels.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
vathreya
Lulu
Posts: 35
Joined: Tue Jun 20, 2017 1:26 am

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by vathreya »

I think it was mostly the strength of schedule metric that did us (Washington B) in. I think this serves as a double-edged sword, because it helps teams in strong regions that don’t put up good statistics due to strong opponents, but also hurts teams in relatively weak regions that put up relatively good statistics. I don’t have a solution here, but I think it might be worth discussing the fact that teams in relatively weak regions basically have to destroy the field in order to qualify, setting a somewhat unnecessarily high bar for these weaker regions that are often comprised of relatively new teams. Is it really worth it to set such a high bar for these teams?

One argument I can see being made is that teams in stronger regions are more likely to accept invitations to nationals, but should this really be the priority?

(I apologize if this comes across as overly aggressive - I’m just trying to start some discussion here).
Vikshar Athreya
UW'22
person361
Lulu
Posts: 51
Joined: Fri Sep 28, 2018 12:18 am

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by person361 »

Some kind of strength of schedule correction is definitely necessary. It's way easier to score points against a weak field than a strong field. I personally think that the current SOS formula is, in general, extremely fair and does the right amount of correction.

However, in this case Washington B's SOS was affected by the fact that 1/5 of their opponents had a PPB of 2.86. This had a disproportionate impact on the PPB_OPP stat. I think that in future years there should be some kind of lower bound on an opponent's PPB for the purposes of SOS, say 7.50. For teams near qualification, the result of a game against a 2.86-PPB team would not be much different than one against a 7.50-PPB team. I think everyone can agree that swapping a 18.00-PPB opponent for a 22.50-PPB opponent has a much larger game impact than swapping a 3.00-PPB opponent for a 7.50-PPB opponent, even though their effect on PPB_OPP is the same.

(There is a larger difference in TUPPTUH between 18.00 and 22.50 vs. between 3.00 and 7.50, so the 18.00-22.50 switch would still have a bigger effect on SOS. But it feels like the difference isn't enough.)
Eric Lu
Montgomery Blair HS '16
MIT '20
Wartortullian
Rikku
Posts: 376
Joined: Mon Dec 19, 2016 10:02 pm
Location: New Haven, CT
Contact:

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by Wartortullian »

After spending a lot of time looking at D-values, reading arguments about DII corrections and PPB clamping, and spilling ink untold on the forums and Discord, there's something I've started to consider:

Why not just use PPB?

Given the massive amount of edge cases involved in deriving a D-value formula, is it really worth it to ignore the teams that we know are getting bubbled due to quirks of the math, in favor of some hypothetical team that might be significantly better at tossups that bonuses?
Matt
jonah
Auron
Posts: 2383
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by jonah »

A Very Long Math Tossup wrote: Wed Feb 12, 2020 3:22 am After spending a lot of time looking at D-values, reading arguments about DII corrections and PPB clamping, and spilling ink untold on the forums and Discord, there's something I've started to consider:

Why not just use PPB?

Given the massive amount of edge cases involved in deriving a D-value formula, is it really worth it to ignore the teams that we know are getting bubbled due to quirks of the math, in favor of some hypothetical team that might be significantly better at tossups that bonuses?
I'm not prepared (in terms of knowledge or time) to engage in a full-throated defense of D-Values, but PPB has edge cases too. For example, a team might lose every game, answer only one tossup all day, sweep the bonus, and thus get 30 PPB that doesn't really reflect their ability at all.

Purely using PPB also allows the possibility of teams qualifying in a different order than they finished on-site. For example, a sixth-place team could have a higher PPB than, and therefore qualify to the ICT ahead of, a first-place team. Reasonable people might disagree on whether that's a problem, but it is to me and (I think) to a lot of NAQT members who care about these issues.

It simply does not sit right with me to throw out half the game (i.e., tossups) in determining who qualifies for a national championship. I believe that sentiment is also widely shared within NAQT. Tonight or soon I'll see if I can do some looking into whether tossup or bonus performance better correlates with actually winning a game, but the conventional wisdom seems to be that tossup performance does.
Jonah Greenthal
National Academic Quiz Tournaments
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by theMoMA »

PPB also correlates much more poorly to ICT performance than D value does. There's nothing hypothetical about that.

A broader point: "why not just use PPB" is a strange argument that brings to bear no data. I don't think anyone is opposed to improving the qualification system, either for Nationals or ICT, but I'll direct you to a post I made about two years ago for the broad proposition that it's not helpful to propose a vast overhaul to the system based on a hunch. The data are out there, and if you would like to take a shot at figuring out a new way to rank teams from different sites, there's nothing stopping you from doing it. Or, if your sentiments are more in the reform-the-system vein, people have demonstrated that they are usually responsive to well-considered critiques of how the qualification system breaks down in certain cases. For instance, based on my experience with the calculations and regression analysis, I think it would be smart to investigate whether a lower bound on opponent PPB would improve the performance of D/A value, based on Eric's post above.

I'm struck by the phrasing "teams that we know are getting bubbled due to quirks of the math." Simply put, it's very difficult to compare similarly performing teams at different sites, playing opponents of different strengths, on questions that are easier than the difficulty of the championship tournament. Regardless of what statistics you use, whether it's raw PPB or something more involved, I'd submit that it's impossible to avoid the problem of, say, inviting the statistically 22nd-best team over the statistically 23rd-best team, when the 23rd-best team is actually more skilled at the championship level. In a system with a limited number of championship invitations based on performance at different sites on questions of lower difficulty, the cutoff has to be somewhere, and the qualification measure will probably not have such great resolving power as to perfectly distinguish teams near the cutoff. Given the vagaries of quizbowl performance, even testing for "perfect distinction" using ICT data would be almost impossible.

Because I've run the numbers and know that PPB is much less predictive of ICT results than D value is, I submit that changing to a PPB-based qualification system would only exacerbate the issue of deserving teams getting "bubbled."

My response to this whole question of qualification, as I've set forth in the post I linked above, is to continue empirical work on the system we already know is doing a good job predicting team performance at the championship level, so that it can do an even better job of distinguishing teams in the future. As I've said, I'm entirely open to the idea that there's another, better way to use the same data to achieve the same purpose. But I don't think throwing out ideas that are demonstrably poorer at predicting team performance without any data to support your suggestion is a serious attempt at this.
Andrew Hart
Minnesota alum
jonah
Auron
Posts: 2383
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: A Modest Proposal Regarding D2 ICT Qualification

Post by jonah »

Also, I'll reiterate a prior offer that if you want to examine NAQT performance data in some way that's not convenient on our website, talk to me and I'll make reasonable efforts to provide reasonable data in a useful form.
Jonah Greenthal
National Academic Quiz Tournaments
Locked