ICT bracketing discussion

Old college threads.
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

ICT bracketing discussion

Post by Important Bird Area »

Sounds like there's some interest in having a thread about this.

(No question content here; if you'd like to discuss the questions and not the tournament logistics, please request admission to the private forum.)
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

First post: theory of ICT bracketing.

There are three steps to this process, and I suspect that there are no real alternatives to these.

Step 1) SCT data. Our invitations to ICT are based on collecting a bunch of data about how well teams performed at SCT.

Step 2) Roster changes. Teams that register for ICT have the opportunity to submit their rosters through naqt.com. If those rosters differ significantly from the ones that played SCT, I adjust the brackets accordingly (where needed, using the abundance of non-NAQT tournament statistics as guidance).

Step 3) Host bids. Those schools that hosted SCT will have little or no data from SCT. These teams are then added to the rankings to the best of our ability (again, using non-NAQT data as needed).

NAQT does not normally release seeding data to the public, but in this case I will make an exception, since there is interest in public discussion (and the top half or so of ICT seeding is readily deducible from the schedule anyway). Discussion of specific teams to follow after lunch.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: ICT bracketing discussion

Post by ThisIsMyUsername »

I was going to roll all of my (numerous) complaints about this year's ICT into one post (bound together by the obvious theme: that NAQT appears to be nearly entirely out of touch with most aspects of the current collegiate circuit), but I see that things have been separated into different threads, so I'll start here and then move on to the other threads.

The seeding for this tournament was positively atrocious, as the tournament results bear out. The choice of the initial top seeds within each bracket was wrong. The choice of which four teams, from the remaining teams, should be the second seed within each bracket was wrong. The teams that finished 3rd, 4th, 9th, and 10th in the overall rankings and which performed 2nd, 3rd, 6th, and 9th in terms of prelims PPB (i.e. Chicago A, Penn A, Harvard, and MCTC) were placed in the same bracket. In other words, three teams that should all clearly have been in the top bracket and the team that was the top contender from the second bracket to break into the top bracket were seeded in the same prelim bracket!

It would be one thing if these results were totally unpredictable. But, in fact, pretty much everybody plugged into the current circuit knew the relative strength of all these teams. When the brackets were revealed on Friday night, there was a flurry of texts, e-mails, and IRC chatter bemoaning how poorly executed these brackets were. For goodness' sake, Peter Cordeiro-- whom the community at large (justifiably, if unkindly) mocks for his general lack of awareness of how the game works and for his basic inability to understand how to use stats to determine team strength or even perceive current rosters-- predicted the strength of these teams with more accuracy than did NAQT! Either little attempt was made to escape the obvious flaws inherent in using D-value as the primary metric, or that attempt was made with less knowledge or competence than that possessed by even the most fringe members of the current circuit.

I would like an explanation of exactly how this tournament was seeded this time, step-by-step. I would like some assurance that ICT will be seeded in future in some different way that actually sensibly accounts for field strength. And I hope we're going to have a proper discussion now of how to reform this.

I would also like to add that if NAQT decides to continue to use D-value as the primary means of seeding (which I hope they do not), they should be far more transparent about this fact, so that teams can take that into account when forming their Sectionals teams. Right now, all the teams I've been on have considered only how best to qualify X # of teams for ICT. But if this is going to largely determine our seeding, then I and my future team captains need to consider in future how best to balance qualifying X # of teams for ICT and jockeying for a non-terrible prelim bracket for ICT.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Gonzagapuma1
Tidus
Posts: 543
Joined: Tue May 02, 2006 5:02 pm
Location: Chicago, IL

Re: ICT bracketing discussion

Post by Gonzagapuma1 »

I would add that the seeding this year seems to give a disincentive for teams to host SCT, since they may get screwed like Harvard did.
Dan Puma
Alum: Gonzaga, Montgomery College, University of Maryland
Currently Unaffiliated
MAQT President-ish, 2014-2015
Champion of the Modern World, 2014
Former Member, PACE
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

ThisIsMyUsername wrote:I would like an explanation of exactly how this tournament was seeded this time, step-by-step.
OK, let's go. Here's the top half of the ICT seeding, with, where necessary, comments on why I seeded them where I did.

#1 seeds

Virginia (#1 overall seed, won the tournament)
Penn A
Michigan
Maryland A (probably a shade high; we didn't know that Arun wasn't attending ICT until after the schedules had been printed)

#2 seeds

Yale
Reynolds
Chicago A (We didn't know Max was playing Division I, because Chicago never submitted its rosters to NAQT.)
Stanford (I overrated this team, probably because I had seen them play a lot in person.)

#3 seeds

Illinois
Harvard (This is an obvious underseed; they hosted SCT)
Georgia Tech
Ohio State

#4 seeds

Chicago B
MIT
Michigan State
Columbia

Teams that I overlooked but made it into the top 16 anyway:

Minneapolis CTC
Maryland B
Cornell

(the three teams seeded 1-16 but finished 17-32: Michigan State, Chicago B, Stanford)
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

And finally, here's a large theoretical question (one we probably won't solve by 2015, but I would be interested to hear people's thoughts):

If D-values aren't accurately predicting ICT performance, can we, as a community, construct a better statistic? And: should we consider changing how teams qualify for ICT in the first place?
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Adventure Temple Trail
Auron
Posts: 2754
Joined: Tue Jul 15, 2008 9:52 pm

Re: ICT bracketing discussion

Post by Adventure Temple Trail »

Boo on NAQT for ignoring MCTC outright; that makes basically no sense given Rob and Bernadette's past playing history and how well they did at SCT in numerical terms.
bt_green_warbler wrote:should we consider changing how teams qualify for ICT in the first place?
I guess it would be nice if winning a regular-difficulty or higher tournament before SCT outright automatically qualified your team for ICT; that would remove a lot of stress for the upper third or so of the Division I field, who could more or less start planning their flights by mid-October, and compel NAQT to pay closer attention to non-NAQT statistics and matchups. But it's not super pressing from what I can tell to do this; the seeding just needs to be done by looking at those non-NAQT tournaments more closely regardless.
Matt Jackson
University of Chicago '24
Yale '14, Georgetown Day School '10
member emeritus, ACF
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ICT bracketing discussion

Post by grapesmoker »

bt_green_warbler wrote:And finally, here's a large theoretical question (one we probably won't solve by 2015, but I would be interested to hear people's thoughts):

If D-values aren't accurately predicting ICT performance, can we, as a community, construct a better statistic? And: should we consider changing how teams qualify for ICT in the first place?
At some point, any statistic includes some assumptions about what goes into it, which may not be any better than the "eyeball test" for a good seed. I realize that having An Official Statistic means that there's a number you can point to, but I think the utility of such metrics is questionable. I think the best thing to do is just look at performance in many regular-season events, and use your best judgment to seed accordingly.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

Some reasons that we place more weight on SCT than on non-NAQT tournaments:

-differences in distribution (real and can very much alter seeding given the sort of fine distinctions we're talking about here)

-non-NAQT events are often further removed from ICT in time

-and for both of the above reasons often have distinct rosters from those attending ICT
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: ICT bracketing discussion

Post by Peter13 »

I heard my name get dropped in. Just a quick comment to John, I understand I made a mistake, said things that were unequivocally, unimaginably wrong and based on outdated data, and then went on a binge on qbwiki and hdwhite.org/qb, and other recommended sites, and hopefully rebuilding whatever Rep that anyone will give me.
Now on to ICT...

I see several odd choices as to why some teams were grouped the way they were. Harvard had a very hard group to make it to the top bracket (as stated above), while Reynold CC's was much easier. Michigan State would have to get more than 250 points per game to get into the third bracket, while Western Kentucky squeaked in with around 140 points a game. One lucky game seemed to make a lot of a difference, especially in the middle of the field. I jokingly said Michigan State wouldn't do as well as expected, but I believe very few people would think they would get 25th. One could say formatting was an issue, but the bracketing seemed to be a big issue. After the final games were played I was asking myself "How is Illonois in the top bracket and Harvard isn't." The answer is bad bracketing, ultimately leading to one lucky or unlucky game. And as John clearly pointed out, if people like me, or even on a more outer fringe can get this, why cannot people who know the general strength of team much better, and deal with this on a pretty well everyday basis.

I appreciate what NAQT does, but when I see a problem, all I can think is how to make the thing I love even better. To answer Jeff's question, I don't have a perfect answer. But what I would do (some of which is already mentioned) is look at some of the more academic non-NAQT tournaments. Calculate each players points per game by themselves, and compare them to their teammates. Compare teams who are in mirrored tournaments by PPBs. Look at rankings done by other people, especially Mike Cheyne's. There are probably other things, but I can't think to name them now.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team
User avatar
Harpie's Feather Duster
Forums Staff: Administrator
Posts: 1117
Joined: Thu Feb 10, 2011 11:45 pm
Location: Chicago

Re: ICT bracketing discussion

Post by Harpie's Feather Duster »

In contrast to the mistakes in D1 seeding, I thought the D2 seeding was done very fairly and seemed to reflect the results fairly accurately other than probable underseeding of McGill.
Dylan Minarik

Hamburger University 'XX
Northwestern '17
Belvidere North High School '13

Member Emeritus, PACE

JRPG Champion, BACK TO BACK Robot Slayer
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

For what it's worth, Division II is always significantly easier to bracket (mostly because there is much less roster change from SCT).
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ICT bracketing discussion

Post by grapesmoker »

bt_green_warbler wrote:Some reasons that we place more weight on SCT than on non-NAQT tournaments:

-differences in distribution (real and can very much alter seeding given the sort of fine distinctions we're talking about here)

-non-NAQT events are often further removed from ICT in time

-and for both of the above reasons often have distinct rosters from those attending ICT
Sure, there are definitely differences between ICT and other tournaments. However, historically, the good teams at ICT are the good teams at Nationals are the good teams at pretty much all the other tournaments. So while the distributional difference has some impact, I think it mostly serves to permute teams around the same rank that they would have anyway, rather than turn a top-bracket team into a 4-seed. The time issue I don't think is that significant; barring major upheavals, you mostly know who your team is going to be by the end of the fall, and in any case, you had several medium to hard events in the spring, including Regionals and CRR. Rosters change, but not that much, and you can usually find out how the people on the roster played.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
Fado Alexandrino
Yuna
Posts: 834
Joined: Sat Jun 12, 2010 8:46 pm
Location: Farhaven, Ontario

Re: ICT bracketing discussion

Post by Fado Alexandrino »

Golden-bellied Starfrontlet wrote:In contrast to the mistakes in D1 seeding, I thought the D2 seeding was done very fairly and seemed to reflect the results fairly accurately other than probable underseeding of McGill.
I don't think McGill was particularly underseeded compared to lets say, Yale given as their group's second seed. Some things went right for us, such as the Chicago game having not one, but two distinctively Canadian tossups. Does NAQT underseed Canadian teams because of the inherent subject imbalance under the assumption we won't be very competitive compared to the American teams on American stuff?

However, one comment: After our loss to Columbia we (McGill) were upset we lost to a team that we wrote noted had a lower D-value and lower Sectionals ppb (with the same roster), but I did notice that Columbia had a higher power rate than we did both at that moment at ICT as well as at their SCT. If not already brought up over the years, I think a value associated with the strength of schedule multiplied by some function of power should be included with the D value. In hindsight, Harvard, McGill and Columbia should not have been in the same round robin, but based on just overall D-value, there wasn't an obvious problem.

Jeff, could you provide the same seeding analysis for division 2 as you provided for division 1?
Thank you.
Joe Su, OCT
Lisgar 2012, McGill 2015, McGill 2019, Queen's 2020
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: ICT bracketing discussion

Post by ThisIsMyUsername »

Jeff's post is pretty shocking:
bt_green_warbler wrote: OK, let's go. Here's the top half of the ICT seeding, with, where necessary, comments on why I seeded them where I did.
Um, actually some comments as to why you seeded both Michigan and Maryland A ahead of Yale (in the latter case, intervening against D-value since Yale outranked Maryland A) are probably necessary too.
Chicago A (We didn't know Max was playing Division I, because Chicago never submitted its rosters to NAQT.)
So, I am not the president of this club, and I never will be in my time here since that is an undergraduate officer. The only time I have tried to check or alter a roster on NAQT's website, the website informed me that I was not authorized to do so. I do not know why our president failed to submit a roster to you. I'll reprimand her when I next talk to her. But I had no way of checking whether our roster had been submitted, and was never informed that it hadn't been.

That being said, I'm pretty sure that every single one of our opponents knew that Max was playing DI, and I told Seth and Selene our roster when I had dinner with them in February, so they presumably could have told you if you had asked them.

So, if you're the person in charge of drawing up the brackets for one of the two most important quizbowl tournaments of they year, which uses a format that's extremely sensitive to the initial bracketing, and you don't have a roster for a team whose seeding will obviously have a large effect on who makes the top bracket, which one of these courses of action seems like a good idea?:

(A) E-mailing some member of the team to ask them what their roster is
(B) Checking with some of your colleagues at NAQT who are more plugged in to that region to ask if they know what that team's roster is
(C) Checking the forum of which you are the administrator to see if that information is readily available to all (as indeed it was)
(D) Some combination of one of more of the above-listed options
(E) Guessing
Stanford (I overrated this team, probably because I had seen them play a lot in person.)
Stanford ranked 15th in D-value from their SCT results, and finished 18th in the overall standings at ICT. You seeded them 8th, in the top bracket, ahead of 6 teams (i.e. 3/4 of a bracket worth of teams) all of whom had exceeded Stanford's PPB at SCT. The explanation you have given for why you did this is: "I had seen them play a lot in person". Jeff, you are one of my favorite individuals in quizbowl, as a player and as a person, so I'm going to take this opportunity to pause for a moment and ask you if there were any other reasons you seeded them as you did, before I react to this in print.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

The D-value formula already incorporates powers; we have no plan to double-count them.

I would expect that issues of Canadian vs. American content would already be present in the stats from SCT.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

Maryland vs. Yale: I thought Arun was playing ICT when I seeded them (he did not play SCT). Note that the exact order of these two didn't matter because they were 4/5.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: ICT bracketing discussion

Post by Rococo A Go Go »

Even I knew that Max was playing D1 beforehand, and it seems pretty obvious from other results that Harvard and MCTC are great teams this year. I don't want to bitch about our pool since we weren't beating any of the alternatives most likely, but there were times on Saturday that I just found it incredibly shocking that all of those teams were in he same pool. I did at least get to meet and play against a bunch of great players and teams, but that's hardly a good excuse for how it happened.
Peter13 wrote: Michigan State would have to get more than 250 points per game to get into the third bracket, while Western Kentucky squeaked in with around 140 points a game.
I'm still not sure you know how quizbowl tournaments work; I love stats and know they're important, but we play the games for a reason. We beat Michigan State head-to-head, which led to MSU getting stuck in the bottom bracket, and I guess statistically we didn't deserve to or whatever, but we did it.
Nicholas C
KQBA member
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

ThisIsMyUsername wrote:
Stanford (I overrated this team, probably because I had seen them play a lot in person.)
Stanford ranked 15th in D-value from their SCT results, and finished 18th in the overall standings at ICT. You seeded them 8th, in the top bracket, ahead of 6 teams (i.e. 3/4 of a bracket worth of teams) all of whom had exceeded Stanford's PPB at SCT. The explanation you have given for why you did this is: "I had seen them play a lot in person". Jeff, you are one of my favorite individuals in quizbowl, as a player and as a person, so I'm going to take this opportunity to pause for a moment and ask you if there were any other reasons you seeded them as you did, before I react to this in print.
Stanford ranked 15th in D-value... with an empty chair at SCT instead of their second-best player (Nikhil Desai).

Eighth was too high and I have apologized, but it would have been in no way unreasonable to seed them somewhere in the third tier of rankings.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ICT bracketing discussion

Post by grapesmoker »

It seems to me that because of the relatively smaller sample size of the prelims, any bracketing mistake is exacerbated. With an 8-team bracket, being stuck with two other top-bracket teams can doom you even if you might have made the top bracket yourself.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: ICT bracketing discussion

Post by Peter13 »

As a Canadian, I have to say that usually Americans have better grasp at American questions and Canadians dominate Canadian questions. Since at SCT, only Canadians are playing American questions, the top teams have a little more time (perhaps an extra clue or two) to answer, versus against an American team, where that would not necessarily be the case. Based on current distributions, I think you can guess which is more important at ICT.

To answer Nick's rebuttal, I picked that example because it had a huge disparity. Your win against Michigan State was too close for me to discern that you were better, it might of just been a lucky/unlucky game. This was WKU's only game within 100 points, while MSU had three such games. They were better from every other major statistic (PPG, PPB, P/N, etc.) From my lowly opinion, I feel they should have went through to the third bracket.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team
User avatar
Cheynem
Sin
Posts: 7220
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: ICT bracketing discussion

Post by Cheynem »

Yeah, one other thing complicating bracketing (which doesn't include the wonky stuff at the top) is that this was an excellent field. From about team 13 (in the second bracket) to about team 28 or so (in the last bracket), you have a LOT of teams with similar bonus conversions and PPG's. This means the nuance from a 3 seed to a 7 (or 8!) seed could be very challenging to configure, and as Jerry says, with a small margin for error, one upset loss could drop you a full bracket lower.

Possibly concrete options for the future:

1. Larger prelim brackets?
2. Force teams to submit (and lock in) rosters a month (or weeks?) beforehand.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
gyre and gimble
Yuna
Posts: 765
Joined: Tue Feb 17, 2009 2:45 am

Re: ICT bracketing discussion

Post by gyre and gimble »

Not checking for rosters and then judging based on limited knowledge is obviously unacceptable, but beyond that, D-values are not the only place where information is available. Jacob O'Rourke has been maintaining a very comprehensive summary of results from regular or higher difficulty events throughout the year complete with rankings in PPB. In the case of Harvard's seeding, D-value says nothing because we didn't play (and we really wanted to by the way, so suffice it to say I will never offer to host SCT again wherever I end up at school in the future). So you look at other results, which tell you we outperformed Stanford (by huge margins) and Illinois (by closer but still consistently significant margins) on every single set this year. I don't know how you could possibly justify sticking us where we were. Also, clearly MCTC's D-value was severely limited by SOS. You should have realized this and also remembered that they finished 11th last year before slapping them with a 7th seed.
Last edited by gyre and gimble on Mon Mar 31, 2014 4:29 pm, edited 1 time in total.
Stephen Liu
Torrey Pines '10
Harvard '14
Stanford '17
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

It's not reasonable to carry out all three of those steps for all 64 teams. It is the team's responsibility to enter their rosters into our registration system.

Edited to clarify that we do email teams asking for roster updates --JTH
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

Stephen: my apologies for the underseeding of Harvard; you and your teammates deserved an opportunity to compete for a higher place in the ICT field.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: ICT bracketing discussion

Post by Rococo A Go Go »

Cheynem wrote: 2. Force teams to submit (and lock in) rosters a month (or weeks?) beforehand.
Sometimes people do back out a few days before the tournament. It happened to us this year (I don't know if that affected our seeding at all, results say maybe it should have at least) and I understand that rosters can change multiple times leading up to the tournament.

I think I could live with locking rosters in 5 days beforehand or something like that though.
Nicholas C
KQBA member
User avatar
Cheynem
Sin
Posts: 7220
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: ICT bracketing discussion

Post by Cheynem »

Well, obviously things could happen, I don't mean that if your roster doesn't end up matching that you are kicked out or something, but just requiring them to be entered (with hopefully no attempts at "gaming the roster system" with bad faith stuff) might be a good thing.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
Adventure Temple Trail
Auron
Posts: 2754
Joined: Tue Jul 15, 2008 9:52 pm

Re: ICT bracketing discussion

Post by Adventure Temple Trail »

Cheynem wrote:Possibly concrete options for the future:

1. Larger prelim brackets?
2. Force teams to submit (and lock in) rosters a month (or weeks?) beforehand.
Both of these are good ideas. Re: 1: especially if ICT has a huge surplus of available staff again next year, upping the Division I field to 36 teams seems possible. But I'm more interested in 2, which seems to have garnered this response
bt_green_warbler wrote:Contacting all 64 teams is not feasible. It is the team's responsibility to enter their rosters into our registration system.
As I remember it, R. sent out a reminder to teams asking for roster checks/changes/updates between SCT and ICT as the third point among five in an update to "Team Contacts" on March 13th. So, as a zeroth point, it seems eminently possible to contact all the team-listed email addresses in one burst. (If we're a bit more adventurous, and want to make targeted emails, it probably takes all of two hours to copy/paste each team contact's first name into a form email reminding them to update rosters. Or you could even write a script that does so automatically.) But more constructively: it definitely seems possible to
-isolate the email about roster changes/updates, so teams pay more attention to it instead of glossing over it in a list of five
-add some text in bold in said isolated email, saying it's super important to present us a correct roster, or else you may be improperly seeded etc. etc.
-require all teams to update their roster by two weeks before the tournament, upon penalty of (fee increase? dropping them?)
with very little additional effort, which would all decrease the likelihood that anyone forgets to update their information.
Matt Jackson
University of Chicago '24
Yale '14, Georgetown Day School '10
member emeritus, ACF
User avatar
AKKOLADE
Sin
Posts: 15783
Joined: Thu Apr 24, 2003 8:08 am

Re: ICT bracketing discussion

Post by AKKOLADE »

bt_green_warbler wrote:Contacting all 64 teams is not feasible. It is the team's responsibility to enter their rosters into our registration system.
Ehhhhh, you can email 64 teams the week before the tournament.
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

Cheynem wrote:1. Larger prelim brackets?
2. Force teams to submit (and lock in) rosters a month (or weeks?) beforehand.
(1) is unsustainable without moving back to a two-day ICT

(2) is kind of useless without sanctions, and it would be silly to actually sanction teams that had last-minute roster changes.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2891
Joined: Sun Nov 15, 2009 12:57 am

Re: ICT bracketing discussion

Post by Cody »

Getting rosters are an important component of the solution, but the real solution is to: (a) have a seeding meeting/discussion that (b) that invites some non-NAQT people who know things about the circuit. Seeds don't necessarily need to be determined here, but having such a meeting would help alleviate the problem of one person/NAQT people missing things.* And, of course, scrapping any notion that D-value means anything about ICT performance in a broad sense.

*I don't actually blame NAQT for missing things like Chicago A having Max: many of their members are not actively participating in the wider college circuit and I only knew about this from IRC, for instance. However, as a consequence, better practices should be in place so that they don't need to have a lot of members actively participating in the wider college circuit (e.g. by bringing in outside people to help with seeding).
Last edited by Cody on Mon Mar 31, 2014 4:43 pm, edited 2 times in total.
Cody Voight, VCU ’14.
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: ICT bracketing discussion

Post by ThisIsMyUsername »

bt_green_warbler wrote:
ThisIsMyUsername wrote:
Stanford (I overrated this team, probably because I had seen them play a lot in person.)
Stanford ranked 15th in D-value from their SCT results, and finished 18th in the overall standings at ICT. You seeded them 8th, in the top bracket, ahead of 6 teams (i.e. 3/4 of a bracket worth of teams) all of whom had exceeded Stanford's PPB at SCT. The explanation you have given for why you did this is: "I had seen them play a lot in person". Jeff, you are one of my favorite individuals in quizbowl, as a player and as a person, so I'm going to take this opportunity to pause for a moment and ask you if there were any other reasons you seeded them as you did, before I react to this in print.
Stanford ranked 15th in D-value... with an empty chair at SCT instead of their second-best player (Nikhil Desai).

Eighth was too high and I have apologized, but it would have been in no way unreasonable to seed them somewhere in the third tier of rankings.
Jeff, the point of my needling you now (and to me, the very point of this thread's existence) is not to extract further apologies from you. The point is to find out how you made the changes to the D-values that you did. You are currently giving the (hopefully false) impression that Stanford's too-high position in the seedings was based on your personal witness of their performances, rather on some sort of stats or results-based metric. If this were actually the case (and NAQT fills in the data gaps in their D-values with guesswork), people would be pretty justified in being angry, given the obvious importance of initial seedings. However, your original post suggests that you attempted to account for roster changes or lack of SCT data (due to host bids) using stats from non-NAQT data, which is obviously the right thing to do, and which should have made any inevitable regional bias you might have largely irrelevant. If you could tell us what data you looked at and how you employed it to determine where exactly to seed teams (i.e. how you combined it with D-values), we could offer constructive suggestions as to how to improve the seeding methods for future (since the community at large had access to same data as you and generated vastly more accurate predictions). Absent that, this thread is probably not going to generate much productive feedback that can help you in future rather than just bitterness at this year's bad brackets.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

Stanford rosters:

Penn Bowl: Benji, Roger, Thomas, Nikhil
ACF Fall: James, Benji, Roger, Dennis
SCT: James, Benji, Thomas
Submit: Benji, James, Thomas, Ian
Cane Ridge: Benji, Nikhil

Note that there are no 2013-14 tournaments with James, Nikhil, and Benji all on the same team, so any exact data would of necessity have been supplemented by estimation.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: ICT bracketing discussion

Post by ThisIsMyUsername »

bt_green_warbler wrote:Stanford rosters:

Penn Bowl: Benji, Roger, Thomas, Nikhil
ACF Fall: James, Benji, Roger, Dennis
SCT: James, Benji, Thomas
Submit: Benji, James, Thomas, Ian
Cane Ridge: Benji, Nikhil

Note that there are no 2013-14 tournaments with James, Nikhil, and Benji all on the same team, so any exact data would of necessity have been supplemented by estimation.
Sorry, if I was unclear; I didn't mean: "How did you account for missing data in Stanford's case specifically?"; I meant: "Could you please concretely explain how non-D-value data is used to help seed teams?" i.e.: Is non-NAQT data used only in cases where there are missing rosters or autobids or is it generally used to supplement D-values? How is outside data actually used/weighted? Do you run a D-value like metric on non-NAQT tournaments to generate data? Or are you just looking at PPB / PPG / powers, etc.? Is this weighted, or do you just used the outside data as a corrective in cases where it seems radically different from what D-values are telling you?

The other (potential) problem with my question is that it assumes that D-value is in some way a good base metric to work off of to begin with. Has someone examined the data connecting SCT D-values to ICT finishes across the years to see if D-value is actually a good predictor? If it is generally, but there are some exceptions, then perhaps we can look at the exceptions and look for methods that might have been used to predict them so we could apply them in future? If D-value has not predicted finishes well, can we please, please stop using it to determine seedings?

I would like to note that the problem with the "personal responsibility" doctrine (or even the sanction-imposition suggested by someone else) for roster reporting is that due to the nature of the tournament format, all teams in contention for the top bracket are affected by the seedings established by the available roster information. So, such a system makes the entire field reliant on everyone doing their part. Chicago A was not the team that was most adversely affected by the fact that our roster wasn't known. My posting is not about our disappointing finish, which was mostly due to our underperformance in the playoffs, and the good performance of our opponents therein. Within our prelim bracket, we were the statistically strongest team and we cleared the bracket, so our bad seeding didn't hurt us much. It was mainly other teams that were screwed over by how we got seeded.

For this reason, I would like to apologize to any teams whose ICT results might have been harmed by Chicago's failure to report our rosters. I did not know that this had happened, and it would never have occurred to me that we would get seeded below the Top 5 (or that NAQT might not know that Max was playing DI). I will never be president of the club, so I cannot personally enter this data for future Chicago teams, but I will do my utmost to make sure this doesn't occur again during my time here.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2891
Joined: Sun Nov 15, 2009 12:57 am

Re: ICT bracketing discussion

Post by Cody »

As far as I am aware, you do not have to be president or even in any organizational role to register/edit teams for NAQT tournaments. I regularly registered VCU teams as Treasurer and other people on the VCU team had access to the NAQT system to edit our registration information even without an organizational role. I believe all that is necessary is that you be affiliated and maybe some other minor requirements.
Cody Voight, VCU ’14.
User avatar
Cheynem
Sin
Posts: 7220
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: ICT bracketing discussion

Post by Cheynem »

You don't need to be president, but if someone else registers the teams, I don't know if you have the ability to edit the teams. I once registered Minnesota's teams despite never holding any position with the club, but I think this freezes out the ability of other people to do the same (?)
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
Bartleby
Rikku
Posts: 315
Joined: Wed Dec 02, 2009 5:45 pm
Location: Philadelphia, PA

Re: ICT bracketing discussion

Post by Bartleby »

Cheynem wrote:You don't need to be president, but if someone else registers the teams, I don't know if you have the ability to edit the teams. I once registered Minnesota's teams despite never holding any position with the club, but I think this freezes out the ability of other people to do the same (?)
This is my understanding of the system as well. You would need to have a communal account, or at the very least have the person who registered share their password in order to make roster updates.
Brian McNamara
Western University '13
University of Waterloo '14
Temple University '20
User avatar
Blackboard Monitor Vimes
Auron
Posts: 2362
Joined: Sat Aug 18, 2007 5:40 pm
Location: Richmond, VA

Re: ICT bracketing discussion

Post by Blackboard Monitor Vimes »

Cheynem wrote:You don't need to be president, but if someone else registers the teams, I don't know if you have the ability to edit the teams. I once registered Minnesota's teams despite never holding any position with the club, but I think this freezes out the ability of other people to do the same (?)
I had Josh register our teams for SCT as part of his president training and subsequently edited the rosters myself when a late change was communicated to me, so this can be done. Pretty sure all you need is an account with NAQT and a listed affiliation with the team; I know if I try to click on a team while not signed in it will tell me I'm not authorized.
Sam L,
Maggie L. Walker Governor's School 2010 / UVA 2014 / VCU School of Education 2016
PACE
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: ICT bracketing discussion

Post by Matt Weiner »

In lieu of going back and quoting lots of little phrases that I think are worth responding to in this thread, I will just offer some aphoristic thoughts on various subjects, not necessarily moral nor diverting, that have been glanced upon above:

*The fundamental issue with seeding is the D-value and a gulf of understanding as to what the D-value is supposed to do. If it is supposed to "be a superior method to completely nontransparent, subjective selection of ICT teams" or "be a superior method to the S-value, which was both less mathematically sensible and less transparent" then it has achieved either/both of those goals. If it is supposed to roughly sort out the group of "teams in the top 50% of DI SCT teams" from "teams in the bottom 50%" then it's fine for that purpose, too. If it is actually supposed to make a reliably correct determination in close cases, such as inviting team 32 to the ICT and waitlisting team 33, or seeding some team 5th and another team 6th, then there is no indication it can achieve this goal. There are two reasons, one easily fixable, one not. The non-fixable one is that hosts of SCTs who expect to be contending teams at ICT and teams changing their roster to a lineup that has never played an NAQT event together before prior to ICT are going to be perpetual problems until such time as the quizbowl economy radically changes, and there is no perfect way to deal with them. The fixable one is that the D-value does a subpar job of dealing with mixed fields/D1 teams playing D2 questions, and severely underrates teams who are the only ICT contender at their SCT. The latter problems seem fixable with some basic statistical analysis -- figure out what the ACTUAL numerical benefit of a D1 team playing mostly D2 opponents on the D2 set is. It's certainly a lot more than the meager 10% penalty that is currently assessed for this and allows teams in that position to get an inflated D-score. Similarly, figure out a way to account for SOS that doesn't leave a team who could have contended at a competitive SCT unable to build up a good D-value no matter what they do to the packets if they find themselves playing mostly lower-tier teams at SCT. Once this is done, there will still be cases caused by the unfixable problems that people will just have to live with. Ultimately, there's 32 spots in the tournament (or 36 or 40 or whatever it may change to in the future) and there will always be a team n + 1 who feels slighted, and the answer can't be anything more than "everything that could be done to get this right was done, if you didn't want to land in spot 33 you should have scored more points at Sectionals," and similar for seeding ICT within those teams who do get invited.

*I don't like that the popular high school fallacy that a team has an unlimited right to have "one lucky or unlucky game" without consequence is filtering up to the college level. This is a competition. Wins and losses are awarded. If you want to advance at the ICT, you will need to beat very good teams to do so. The whole point of getting the D-value and the seeding right is to create the situation where teams control their own destiny by winning or losing games, not to avoid it.

*To expand that note, I think there are some weird ideas about how to determine what a fair or unfair bracket is. The notion that the final order of finish must match the teams' bonus conversion or their poll ranking or whatever is baseless. If it were possible to perfectly construct the brackets based on prior knowledge, then there would be no reason to play the tournament -- the team which we somehow perfectly know is #1 could be mailed the trophy, saving us all much time and expense. It's not just that sometimes "better" teams lose games for fluke reasons, it's also that this weird cult of bonus conversion that has arisen in the last two years totally ignores how quizbowl is played. I played on a team on Saturday that had little to no science knowledge; we had no hope of having an impressive BC because every fifth bonus was a 0 to 10 for us, with very few exceptions. But, we often could play above our level by performing well on the non-science that comprises 80% of the tossups to make up for this. There is so much going on in terms of how knowledge and strategy are employed on tossups that makes any expectation of the pure knowledge allegedly measured by bonuses mapping neatly onto outcomes completely fallacious; I don't think people who can't comprehend this really understand how quizbowl works, especially at high levels.

*ICT is a tournament where a lot of great teams come together, and the pigeonhole principle is in full effect. You can't give everyone a top 2 seed, and teams that we all know are good or could win local tournaments will sometimes be seeded lower. Who, exactly, was Harvard supposed to take the place of? Illinois was (after you collapse the two Penn teams) 8th at D-value this year; Harvard was 10th at ICT last year and lost 2 of its players from that tournament. It's completely rational to pick Illinois over Harvard there. You would have a hard time arguing (especially on knowledge available pre-tournament) that Harvard deserved one of those spots more than my team, or Michigan, or Chicago, either. Sure, Stanford was overseeded, but the first clear choice to replace them would have been MCTC. You can't fit all 16 teams that someone thinks are good into the 8 top spots.

Yes, there are a lot of things that NAQT could do better with bracketing, but the suggestions need to be feasible and rational if they are to gain traction, and should not be based on weird made-up principles or mathematical impossibilities.
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
User avatar
vinteuil
Auron
Posts: 1454
Joined: Sun Oct 23, 2011 12:31 pm

Re: ICT bracketing discussion

Post by vinteuil »

I don't notice a lot of positive suggestions in this thread (sorry if I read straight past someone else suggesting these ideas).

In keeping with John's idea that whoever in NAQT (preferably as many well-informed people as possible!) is doing the bracketing needs to have their ear to the ground, it seems reasonable to me that:
  • People doing bracketing could read the pre- and mid-season poll threads and their accompanying write-ups. This takes very little time.
  • NAQT could/should involve as many recently graduated players who play open tournaments during the season as possible, and have thus seen how these teams work.
The other major issue seems to me to be that it's very difficult for anyone (including NAQT here specifically) to account for roster changes.

Would it be possible for teams to submit partial rosters ("we're not sure about our 4th yet, but") to ameliorate this? A good way to implement this might be to allow individual players from a school to sign themselves up on a team, subject to the registrant's override.
Jacob R., ex-Chicago
Susan
Forums Staff: Administrator
Posts: 1812
Joined: Fri Aug 15, 2003 12:43 am

Re: ICT bracketing discussion

Post by Susan »

Would it be possible/would it make sense to pay someone to help Jeff (and whoever else is handling stuff like ICT rosters) out during Nats season? Getting everyone's rosters sorted is one of those tasks that always takes way longer than it seems like it should (unlike some of the people complaining about how this was handled, I've actually done this for a national tournament!) and perhaps an extra pair of hands would be useful (especially if those hands aren't also doing a bunch of writing for the tournament).

Of course, this will only fix project-management-type problems of not repeatedly hounding teams that fail to respond; it won't fix problems caused by teams that deliberately obscure who's playing for them (which, so far as I know, didn't happen here, but has in the past--please don't do this, people!).
Susan
UChicago alum (AB 2003, PhD 2009)
Member emerita, ACF
User avatar
Frater Taciturnus
Auron
Posts: 2463
Joined: Mon Dec 12, 2005 1:26 pm
Location: Richmond, VA

Re: ICT bracketing discussion

Post by Frater Taciturnus »

vinteuil wrote: Would it be possible for teams to submit partial rosters ("we're not sure about our 4th yet, but") to ameliorate this? A good way to implement this might be to allow individual players from a school to sign themselves up on a team, subject to the registrant's override.
In the current system you can literally update your rosters at any time before the tournament.
Janet Berry
[email protected]
she/they
--------------
J. Sargeant Reynolds CC 2008, 2009, 2014
Virginia Commonwealth 2010, 2011, 2012, 2013,
Douglas Freeman 2005, 2006, 2007
User avatar
Blackboard Monitor Vimes
Auron
Posts: 2362
Joined: Sat Aug 18, 2007 5:40 pm
Location: Richmond, VA

Re: ICT bracketing discussion

Post by Blackboard Monitor Vimes »

vinteuil wrote: Would it be possible for teams to submit partial rosters ("we're not sure about our 4th yet, but") to ameliorate this? A good way to implement this might be to allow individual players from a school to sign themselves up on a team, subject to the registrant's override.
With NAQT's current system, it is possible to put 1-6 people on a team and then edit that at any time. So, in a situation like that, you could list only the three you're sure of or all five and then adjust when possible. I had to edit our SCT rosters 5 or 6 times for various reasons; there doesn't seem to be a limit on it. It's a pretty easy system to use; more people should take the 30 seconds to make an naqt.com account and do so.

I've been asking people for rosters in advance as a TD since about 2009 and have met with varying degrees of success; I think NAQT's system that allows you to easily input people's grades and team's UG status and whatnot is actually quite good. At least some of the responsibility should be on teams to make sure the person registering them is on top of their shit.
Sam L,
Maggie L. Walker Governor's School 2010 / UVA 2014 / VCU School of Education 2016
PACE
User avatar
Blackboard Monitor Vimes
Auron
Posts: 2362
Joined: Sat Aug 18, 2007 5:40 pm
Location: Richmond, VA

Re: ICT bracketing discussion

Post by Blackboard Monitor Vimes »

Susan wrote:Would it be possible/would it make sense to pay someone to help Jeff (and whoever else is handling stuff like ICT rosters) out during Nats season? Getting everyone's rosters sorted is one of those tasks that always takes way longer than it seems like it should (unlike some of the people complaining about how this was handled, I've actually done this for a national tournament!) and perhaps an extra pair of hands would be useful (especially if those hands aren't also doing a bunch of writing for the tournament).

Of course, this will only fix project-management-type problems of not repeatedly hounding teams that fail to respond; it won't fix problems caused by teams that deliberately obscure who's playing for them (which, so far as I know, didn't happen here, but has in the past--please don't do this, people!).
I'll volunteer as an NAQT employee uncomfortable writing at ICT difficulty and willing to be a roster minion for $20 or so if such a person is needed. Between here, Facebook, and the IRC, I have pretty extensive contact reach in addition to my familiarity with the current roster submission system. The rest of my team can testify to my hounding abilities.
Sam L,
Maggie L. Walker Governor's School 2010 / UVA 2014 / VCU School of Education 2016
PACE
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

In reply to John's question about incorporation of non-NAQT data into the seeding system:

In general, where teams played the SCT with the same rosters that are signed up for ICT, I usually don't look at non-NAQT data (that is: we believe that SCT performance is the best predictor of ICT performance, for reasons I explained above but boiling down to "recent tournament on the same distribution").

Where I consult non-NAQT results is for cases of 1) roster additions (trying to measure "how much contribution did this team get from people who didn't play SCT?") and 2) host teams, which may not have played SCT at all (in this case, the intent is to get a sense of "how good is this team compared to the other teams that played the same set?") What I have not yet done- and what NAQT should find the time to do next year- is produce a full statistical model that would incorporate bonus conversion data from non-NAQT events (ACF Regionals is probably the best place to start).
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: ICT bracketing discussion

Post by Important Bird Area »

pandabear555 wrote: Jeff, could you provide the same seeding analysis for division 2 as you provided for division 1?
Thank you.
Division II:

#1 seeds:

Chicago (I mistakenly thought Max was on this team)
Northwestern
Harvard
Illinois

#2 seeds:

Yale
McGill
Ohio State
Virginia

#3 seeds:

Missouri
WUSTL
Columbia
MIT

#4 seeds:

Kansas State
Michigan State
NYU
Valencia A

Top 16 teams that I missed: Kenyon (finished 12th), UC San Diego (finished 15th), Minnesota (finished 16th)

Teams I seeded top-16 but didn't finish there: Kansas State (tied 20th), Valencia A (tied 18th), Michigan State (17th)
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4295
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: ICT bracketing discussion

Post by Auks Ran Ova »

I too would like to echo the support for both some sort of seeding meeting and a decreased reliance on/reevaluation of the calculation of D-value, since despite putting up the seventh-highest SCT bonus conversion with a lineup that finished eleventh at last year's ICT and suffered no changes whatsoever, our team got stuck with...quite the uphill battle.
Rob Carson
University of Minnesota '11, MCTC '??, BHSU forever
Member, ACF
Member emeritus, PACE
Writer and Editor, NAQT
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: ICT bracketing discussion

Post by The Ununtiable Twine »

Ukonvasara wrote:I too would like to echo the support for both some sort of seeding meeting and a decreased reliance on/reevaluation of the calculation of D-value, since despite putting up the seventh-highest SCT bonus conversion with a lineup that finished eleventh at last year's ICT and suffered no changes whatsoever, our team got stuck with...quite the uphill battle.
I'm going to chime in here too - Alabama was in a similar situation last year. Despite ranking 8th in PPB and relatively high in powers, we were given a #4 seed despite returning everyone from a team that finished 12th the previous year. We suffered from field strength concerns at SCT but we did hear the same bonuses as everyone else. I'm not saying we deserved to be on the #2 line but I'm convinced we deserved to be a #3 seed somewhere - instead we got put in a bracket with Penn (finished 3rd overall) / Illinois (finished 4th overall) so we were basically doomed to second bracket.

EDIT:

1.) if I recall the bottom half of our morning pool was the weakest in the field so at times it felt like NAQT was dictating our morning pool placement to us. That particular bracket had a very distinct "hey, you two are going to the top bracket, you two the second, etc." feel to it which was disappointing from the point of view of a competitor. It seems like two of the top four teams in the field come out of the same pool more often than they should. MCTC and Harvard's "uphill battle" was similar to ours in this way.

2.) We put up very similar stats to #3 seeds in 2013 ICT prelims. Perhaps in our case the PPBs were so close that field strength was the deciding factor for NAQT. I'll remark also that our powers per 20 tossups at SCT seems to have been slightly higher than most of the teams that were on the #3 line last year. I may be mistaken but the numbers were definitely similar. In any case I don't think we should penalize a team that finished 8th in PPB with a mid-#4 seed, regardless of what the D-value says.

3.) Just a statistical note for those of you that took a look at the final 2013 ICT stats: I know MCTC 2014 performed better than 2013 Alabama in the prelim competitions, but I have every reason to believe that the statistics of those teams would have been similar had I not been rather ill at 2013 ICT. Not making excuses for underperforming statistically, just saying that untimely illnesses can weaken buzzer skills quite a bit, which it did in my case. It's easy to look at the final ICT stats and say "yeah they deserved their #14 seed" but the stats don't account for my health. If you don't account for my health situation you can be further convinced that the D-value works as far as seeding and that the field strength indicator is absolutely fair, which it is not and hence I don't think the seeding was justified in either situation.
[This paragraph is my basically my basis for calling the MCTC 2014 / Alabama 2013 situations analogous. Perhaps I am being a bit too meticulous, but I can be like this at times. Oh yeah, and MCTC had an additional Harvard to deal with (a team which was a bit stronger than MIT 2013 who were the #3 seed in our pool). Yikes!]
Jake Sundberg
Louisiana, Alabama
retired
jonah
Auron
Posts: 2383
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: ICT bracketing discussion

Post by jonah »

Bartleby wrote:
Cheynem wrote:You don't need to be president, but if someone else registers the teams, I don't know if you have the ability to edit the teams. I once registered Minnesota's teams despite never holding any position with the club, but I think this freezes out the ability of other people to do the same (?)
This is my understanding of the system as well. You would need to have a communal account, or at the very least have the person who registered share their password in order to make roster updates.
You can edit registrations for a given team if and only if you are logged in as a user for whom at least one of the following is true:
  • The user is the official team contact for the team
  • The user has a currently active affiliation with the team's school in one of the following roles: coach, administrator, director of student activities, faculty advisor, unofficial advisor, club president, assistant coach, club officer
  • The user is the tournament director
  • The user has a currently active affiliation with the host organization in one of the following roles: coach, employee, administrator, director of student activities, faculty advisor, unofficial advisor, club president, assistant coach, ACUI regional coordinator, ACUI regional director, ACUI program team, club officer
  • The user works for NAQT in a sufficiently authorized role

It is correct that rosters may contain arbitrarily many players and may be updated arbitrarily often between the team's registration and check-in by anyone meeting the above criteria.
Jonah Greenthal
National Academic Quiz Tournaments
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: ICT bracketing discussion

Post by The Ununtiable Twine »

In addition, NAQT gave Georgia Tech a #3 seed this year by finishing #11 overall in D-value in a combined field.

I believe NAQT gave us a #4 or #5 seed in 2012 by finishing #8 overall in D-value in a (much tougher if there is such a thing) combined field. We also added Harrison Brown to our team for the 2012 ICT. My question is as follows: if you gave Georgia Tech a #3 seed in 2014, why didn't you give us a #1 or #2 seed in 2012?

I think this is a pretty good argument as to why using the D-value as your standard for seeding is just plain silly. NAQT obviously used D-value for seeding GT but some other method for seeding Alabama, don't you think? My question is why, especially since GT's only real accomplishment pre-ICT this year was defeating Alabama and Jordan Palmer and their own B-team at our DRAGOON mirror. [...and beating us this year can hardly be called an accomplishment since we are probably ranked somewhere between 25 and 35 overall on any given day] I'm not bashing GT [just using them as an example to clarify my point], but why does NAQT just hand a team with little to no resume on the year a #3 seed?
Jake Sundberg
Louisiana, Alabama
retired
Locked