The Quizbowl Resource Center

Fred wrote:Saying that "most" housewrites have a roughly equivalent difficulty is, well... wrong. Sorry, but HFT had definite differences from BHSAT and LIST.

But those housewrites are more similar to HSAPQ than they are to NAQT in my opinion.

theflyingdeutschman wrote:

Fred wrote:Saying that "most" housewrites have a roughly equivalent difficulty is, well... wrong. Sorry, but HFT had definite differences from BHSAT and LIST.

But those housewrites are more similar to HSAPQ than they are to NAQT in my opinion.

In terms of format, yes. In terms of difficulty, that's a sticker question.

I'm sure this is obvious, but it will need to be dealt with for this to continue to be helpful. Should we notify you about squads and their composition when there are JV brackets at an event? Hoover A, for example, is a varsity team, but Hoover B is our JV squad, and would be Hoover JV A at any other event. Hoover B is another varsity squad that didn't attend for various reasons. Is there a manner that TDs can designate this to make it helpful for your work, or should we notify you on a case-by-case basis?

Even if I were to implement a system by means of which TDs/coaches/players could indicate the appropriate letter designations of teams, I am doubtful that it would be particularly useful: A lot of information about this sort of thing is already posted on the boards, often to help Fred with his rankings, and I don't think that adding an additional facility by means of which to report this information specifically to me would get very many other people to post this information. I do read the forums regularly, and will act on any relevant information that is posted here. I will probably also continue to draw on information about team strength found in Fred's rankings if he continues to include that information.

As always, if you are browsing through the website and find something amiss, you can click on the "Submit correction/information/error report" found at the bottom of every page and send me information if you don't want to clutter up the forums or whatever. I will act on the information sooner and/or later.

I have made the modification you requested, though it will not be reflected in the rankings until I input stats for another tournament.

Ashvin,

I noticed one error, an understandable one, on your October 5th update.

You accidentally added the PHSAT results for Blair Academy of New Jersey to the Maryland Fall Tournament results of Montgomery Blair High School of Maryland. The error probably occurred because Montgomery Blair is often referred to simply as "Blair", and was so designated at Maryland's Fall Tournament. Still, these records should be disentangled, restoring Montgomery Blair to its earlier 8-4 record while giving Blair Academy of NJ its own separate record.

Edward Powers wrote:Ashvin,

I noticed one error, an understandable one, on your October 5th update.

You accidentally added the PHSAT results for Blair Academy of New Jersey to the Maryland Fall Tournament results of Montgomery Blair High School of Maryland. The error probably occurred because Montgomery Blair is often referred to simply as "Blair", and was so designated at Maryland's Fall Tournament. Still, these records should be disentangled, restoring Montgomery Blair to its earlier 8-4 record while giving Blair Academy of NJ its own separate record.

Fixed.

Why haven't teams that played at UIUC Earlybird been added to the rankings yet?

whitesoxfan wrote:Why haven't teams that played at UIUC Earlybird been added to the rankings yet?

I forgot to mark the Earlybird data as ready for addition to the rankings. Rectified.

Tristan Wiley = Macomb's data doesn't appear to be entered.

whitesoxfan wrote:Tristan Wiley = Macomb's data doesn't appear to be entered.

This is true. I had originally excised all of Tristan's matches because I assumed there were some peculiar circumstances surrounding him being Tristan rather than Macomb, but I guess he must've just been playing unaffiliated or something. They will be re-incorporated in the next update.

While I'm here, I should note that as of now, the rankings are not terribly useful. This is because there are too many disjoint circuits that have not yet had any interaction with each other. Luckily, Harvard Fall seems to be drawing teams from a sufficiently ridiculous number of places that every major circuit east of the Rockies will be unified after HFT happens. Starting then, the rankings will become increasingly relevant.

All MOO and MAGNI sites that have occurred so far have been incorporated into the rankings (along with the Florida Gateway Invitational, a college tournament run on an A-set). This has a few consequences.

First, the rankings are no longer continuously numbered. For example, as of this writing, Ladue A is ranked 4th, while the next team, GDS A, is ranked 6th. This is because there is a college team (in this case, Michigan A) ranked 5th. College teams are of course not displayed in the rankings list, but games played against college teams do affect the rankings.

Second, for those of you who are amused by this sort of curiosity, you may now view the rankings of colleges for which data is currently available. I trust interested college players to put very little stock in the rankings, as the circuit is not yet connected; college teams have even higher lineup variability than high school teams; I didn't really bother to account for weakened house teams; etc.

Also: you can now see a snapshot of the state of connectedness of the circuit. It will be updated every so often, but not necessarily at the same time as the rankings (for technical reasons). As you can see, there are (as of this writing) 23 disjoint circuits. I recognize that the page is not exactly user-friendly (it displays team ID numbers, not team names); I may fix this when I get some time over Thanksgiving.

Is there a way to paper over the non-continuous rankings numbers with continuous rankings numbers? If this ends up being something that high school teams might be able to point to in order to, say, secure funding for Nationals, it's probably better for them to be able to say "We're 25th on this statistical rankings page" than "Well, uh, the number says 58, but there are these invisible teams that I may have miscounted..."

(I don't know anything about programming, but) It seems like it should be possible to have a hidden list of rankings numbers for HS and college together, but output an "HSRanking" column on the high school page which attributes each team a number equal to (number of high school teams ranked above this team + 1). Perhaps the same thing could be done for the college page.

I eagerly await the effects of increased cross-circuit travel on these rankings - they could get interesting and relevant quick.

I have now updated the to-do list with the list of things I plan to do. Your suggestion regarding renumbering the rankings is on this list.

Are you not going to incorporate ACF Novice into these rankings?

Charbroil wrote:Are you not going to incorporate ACF Novice into these rankings?

No. I have literally no way of correlating teams at Collegiate Novice with the teams that will be playing regular college tournaments throughout the year. While I'm usually a fan of having more data, Collegiate Novice data would be so actively unhelpful that I think they would reduce the quality of the rankings.

In a similar vein, I generally do not include novice (or otherwise restricted-eligibility) high school tournaments in the rankings. (I do tend to include JV divisions of tournaments that also have a Varsity division, though, because in those cases it is generally fairly reasonable to assume that if there are e.g. two teams from a school in the varsity division and two in the JV division, that the latter two can be considered "C" and "D" teams.)

(This is also a separate issue from if I will include high school tournaments run on Collegiate Novice in the rankings; if any such tournaments exist, I will.)

And now that Harvard Fall has been added to these rankings, I present you with a curated list of the top high school teams in the nation, according to the rankings. Teams have been removed from the list in cases where they are not linked in to the national circuit that now unifies most of the country.

Fred's ranking of each team (as of 13 Nov 2011 and as of 17 Nov 2011) is given in brackets. Teams with asterisks next to their names have especially unreliable rankings, because they have no losses yet.

1. [3|2] Centennial*
2. [6|10] duPont Manual*
3. [16|23] University School of Nashville*
4. [18|20] Ladue*
5. [20|42] Early College at Guilford
6. [4|4] Rockford Auburn
7. [1|1] Hunter
8. [38|40] Hawken*
9. [40|19] Brookwood
10. [9|11] IMSA
11. [7|7] Dorman
12. [11|8] DCC
13. [13|5] Thomas Jefferson
14. [14|14] Dunbar A
15. [46|48] Northmont
16. [10|12] Loyola
17. [61|65] Dunbar B
18. [12|13] St. Anselm's
19. [98|UR] Walnut Hills
20. [UR|79] Alpharetta
21. [88|93] Pope John Paul II
22. [8|6] Seven Lakes
23. [36|38] Warren G. Harding
24. [63|58] Ezell-Harding
25. [45|47] East Chapel Hill

High-ranked teams that are not linked into the national circuit:
[UR|UR] Detroit Country Day [undefeated at HFT Michigan]
[UR|77] West Des Moines Valley [undefeated at Matt Cvijanovich Memorial]
[25|31] Cistercian B [undefeated at TQBA Kickoff]
[UR|UR] Edmond Memorial B [undefeated at OQBA Tulsa Fall]
[UR|UR] Dunbar D [undefeated at the JV division of Danville Fall]
[UR|UR] William Henry Harrison [undefeated at North White Kickoff]
[60|64] Wayzata [undefeated at Olaf Learnament]



At this time, California (separated into North and South), Minnesota, parts of Texas, Indiana, Oklahoma, parts of Michigan, Iowa, and southern Florida are not connected to the rest of the circuit. I expect all of these besides California and maybe southern Florida (depending on whether or not Ransom Everglades travels, basically) to become linked into the main circuit by March or so.

EDIT: Fred Nov 17 rankings have also been added as the second number in brackets.

The best avenue Iowa will have to get linked to the national circuit is through Bettendorf, which will be at the same tournament as Belvidere North this weekend.

EDIT: The only reasonable chance that comes to mind is at NAQT State, which will probably be a disaster again.

Here are the results of the Edmond Memorial Bulldog Tournament run on IS-109 on Nov. 19, 2011

http://naqt.com/stats/tournament-teams. ... nt_id=3386

Excelsior (smack) wrote:

Charbroil wrote:Are you not going to incorporate ACF Novice into these rankings?

No. I have literally no way of correlating teams at Collegiate Novice with the teams that will be playing regular college tournaments throughout the year. While I'm usually a fan of having more data, Collegiate Novice data would be so actively unhelpful that I think they would reduce the quality of the rankings.

I don't think including these data would be actively unhelpful with regard to high school teams who play ACF Novice, as long as college teams at ACF Novice are simply treated as different entities from the teams with the same names playing unrestricted events (i.e., define teams such that Wash U A can't play ACF Novice, but Wash U Novice A can). With that treatment, data from restricted-eligibility college events would be no worse than a small amount of noise.

On the other hand, I've got a problem with including any college tournaments in your high school team rankings, which I'll discuss in my next post.

There are two really big issues with the way college tournaments are incorporated here.

First, high school teams appear to suffer a ranking penalty for playing college tournaments, if games are weighted by the difficulty of the set they were played on. It's been noted already that college tournaments are overweighted by the current scheme, because they are less similar to nationals than nationals would be but they get higher weight than nationals would. With college tournaments included, however, set difficulty is positively correlated with opponent strength -- college teams are, in principle, the strongest opponents, and they are only faced at the toughest tournaments. Since weight is monotonically increasing in set difficulty, and probability of losing in opponent strength, it follows that weight is positively correlated with probability of losing for high school teams who play college tournaments. A few Ls against college opposition much stronger than you, on questions that don't look much like nationals, lower your ranking many times more than your ranking is raised by a few Ws, on questions that don't look much like nationals, against high school opposition much weaker than you -- even though the latter are theoretically potential nationals opponents and the former are not. In fact, this is probably true to some extent for high school tournaments as well -- better teams on average seek out harder tournaments -- but the effect size should be small; with the inclusion of college tournaments, however, the effect is stark.

The second big issue is one of security. Eligibility rules in some states frown on high school teams playing college tournaments, and those teams who do so anyway often play under fictitious names to avoid repercussions with state bureaucracies. The rankings do not appear to respect this anonymity, but instead go ahead and translate those fictitious names into real ones, eliminating teams' anonymity and leaving them open to eligibility recriminations.

The motivating example for both these concerns is the performance of Rock Bridge A at Missiles of October, where they took several losses to college teams under the original team name of "Roseanne Barr". Rock Bridge A appears vastly underrated by Excelsior in comparison to the top 25 poll or Fred Morlan's top 100, while other top (Missouri) teams are ranked fairly similarly in all these publications, and the driver of Rock Bridge's low ranking seems to be the losses they took against college teams (there are almost no other losses).

Ultimately, I recommend replacing the KRACH probability model at the heart of these rankings with DP-FRIAR in order to both: (a) incorporate all the information contained in scores as opposed to just W-L records; and (b) estimate difficulty from the data, at the packet level, instead of arbitrarily assuming difficulty levels for each set.

EDIT: using, of course, the more canonical logit parametrization for p as found in this post rather than the version given in the initial post.

Update with IHSSBCA kickoffs?

First, high school teams appear to suffer a ranking penalty for playing college tournaments, if games are weighted by the difficulty of the set they were played on. It's been noted already that college tournaments are overweighted by the current scheme, because they are less similar to nationals than nationals would be but they get higher weight than nationals would. With college tournaments included, however, set difficulty is positively correlated with opponent strength -- college teams are, in principle, the strongest opponents, and they are only faced at the toughest tournaments. Since weight is monotonically increasing in set difficulty, and probability of losing in opponent strength, it follows that weight is positively correlated with probability of losing for high school teams who play college tournaments. A few Ls against college opposition much stronger than you, on questions that don't look much like nationals, lower your ranking many times more than your ranking is raised by a few Ws, on questions that don't look much like nationals, against high school opposition much weaker than you -- even though the latter are theoretically potential nationals opponents and the former are not. In fact, this is probably true to some extent for high school tournaments as well -- better teams on average seek out harder tournaments -- but the effect size should be small; with the inclusion of college tournaments, however, the effect is stark.

This is true. The existing implementation of incorporating college tournaments is inadequate, and I am planning on fixing that. My proposed solution: construct two sets of rankings - a "high school ranking" and a "college ranking". Both rankings will use all the data available to me, but will weight them differently.

The high school ranking will maximally weight tournaments that are exclusive to high school teams and are of approximately HSNCT/NSC difficulty. The more a tournament deviates in difficulty from HSNCT/NSC in either direction, the lower the weight given to that tournament. In addition, tournaments that are non-exclusive to high school teams will have their weight reduced to accommodate for the increased strength of competition that one might face there.

On the other hand, the college ranking will maximally weight tournaments that are of approximately DI ICT/ACF Nats difficulty. It seems a fair assumption that these are the most difficult tournaments of the year, excluding summer opens (which are of no interest to me); as such, easier tournaments will have lower weights. I think it may be appropriate to add a weight-reduction to tournaments at which both open and non-open teams play (e.g. MO this year), but I'm not sure about that.

How does that sound? If it turns out that the incorporation of tournaments with college teams damages the rankings to the point that it would be better to do without them, I can certainly do away with them. (High school teams are necessary to link the college circuit pre-ICT/ACF Nats, but not vice versa, except insofar as California links into the circuit sooner if college teams are included [via ICT])

The second big issue is one of security. Eligibility rules in some states frown on high school teams playing college tournaments, and those teams who do so anyway often play under fictitious names to avoid repercussions with state bureaucracies. The rankings do not appear to respect this anonymity, but instead go ahead and translate those fictitious names into real ones, eliminating teams' anonymity and leaving them open to eligibility recriminations.

I was under the impression that playing pseudonymously was a mere formality for the purposes of not technically being affiliated with one's school and that nobody was actually fooled by them. If this is not the case, I will certainly stop doing that.

Ultimately, I recommend replacing the KRACH probability model at the heart of these rankings with DP-FRIAR in order to both: (a) incorporate all the information contained in scores as opposed to just W-L records;

I don't think I actually understand DP-FRIAR. Nonetheless, let me try: We obtain a pair of binomial distributions for Y (total score) and Y_1|Y (a team's score given a particular total score). We sample multiple data points for Y by examining all games played on a particular packet. This much makes sense. How do we then sample multiple data points for Y_1|Y?

In any case, once we've sampled multiple points for Y and Y_1|Y, we can then estimate p based on the distribution for Y; what is the analogous parameter for the Y_1|Y distribution? Is it still ((p_1 - p_2) / 2(p_1*p_2)) as given in your first post, or is it something else?

So once we've determined p and whatever the parameter is for the Y_1|Y distribution, we use the "canonical logit parametrization" for p you gave in the second post to compute r_1 and r_2. These r_1 and r_2 are the ultimate output of the process that I would then use to rank teams.

In order to compare different tournaments, we would then need to use the modified distributions Y | different_set with some value of p_D (which we also determine from the data?).

(b) estimate difficulty from the data, at the packet level, instead of arbitrarily assuming difficulty levels for each set.

How does DP-FRIAR behave without access to packet-level data (that is, we don't know which games occurred on which packets)? Also, does it even work if I don't have access to scores (but instead only win-loss records)?

Do note that I was going to start determining set-level difficulty from PPB data eventually; the arbitrary weight's I've picked so far are just stopgaps for the meantime.



EDIT:

Update with IHSSBCA kickoffs?

Probably not until past New Year's.

Excelsior (smack) wrote:Do note that I was going to start determining set-level difficulty from PPB data eventually; the arbitrary weight's I've picked so far are just stopgaps for the meantime.

I might have gotten an answer for this previously, but I don't recall it... I still don't understand what the motivation is for "harder tournament = more important".

jonpin wrote:

Excelsior (smack) wrote:Do note that I was going to start determining set-level difficulty from PPB data eventually; the arbitrary weight's I've picked so far are just stopgaps for the meantime.

I might have gotten an answer for this previously, but I don't recall it... I still don't understand what the motivation is for "harder tournament = more important".

In theory, once I modify the weightings as mentioned in my previous post (to devalue college tournaments for ranking high school teams), these rankings will in some sense be an approximation of how teams will perform at nationals. We expect performance at nationals to correlate most closely with sets that are most similar to the sets that will be used at nationals. The only quantifiable measure of similarness of sets (that I can think of, anyway) is their difficulty; as such, it seems to me that weighting sets that are of comparable difficulty to HSNCT/NSC the most heavily would yield the most useful rankings.

That said, given various discussions surrounding HFT this year, I am not sure if my position is correct. It is certainly true that there exists some difficulty level below which a better team will not win against a worse team nearly as often as it would on HSNCT/NSC (and consequently, games played on sets below that difficulty level ought to have reduced weight). Empirically speaking, however, I don't know what that difficulty level is. It may well be that everything harder than A-sets have similar odds of victory for national-contending teams as do nationals-level sets; or it might be that even A-sets have similar-enough odds; or maybe we need things harder than IS-sets to get close-enough odds.

(Whatever the case may be, my weights are too extreme right now, I'm pretty sure, even ignoring the college sets that are currently included; I think that a spread of ~0.85 to 1.2 from A-sets to HSNCT/NSC would adequately capture the variation in victory odds across those sets, though this estimate is based only on intuition and not on any actual rigorous analysis of data.)

This doesn't look this has been added: (HFT mirror at WKU)
viewtopic.php?f=1&t=11551&p=230002#p230002

Where Lexington A = Dunbar

He's only added tournaments from November 12th and earlier right now.

It would be nice to see an update to the rankings, since there are a lot of tournaments yet to be entered. I'm sure there will be some drastic changes in the numbers. It'll also help people quickly access stats for teams if they're thinking about voting in the Mid-Season Poll that George is running.

This is still really cool to finally have a comprehensive one of these.

large, visible red letters on the Excelsior web page wrote:The rankings may not be updated until after New Year's, because I am currently in the midst of rewriting the entire site to make it more maintainable and extensible.

They're large, visible, and red!

It's after New Year's. I agree with Mr. Chrzanowski that these rankings are great, and it would be useful if we could get an update to help us vote in the Mid-Season Poll.

Ashvin I will Excelsior (smack) you if you don't update your rankings right now.

Edit: It's been two whole days since the New Year YOU MONSTER

Really i just asked because of George's poll. Relax.

Owing to multiple factors (most of which pertain to me being lazy, bad at coding, and also delinquent on other quizbowl-related duties), the Excelsior Rankings will not be updated in the near future. I have received some suggestions that I put off the site revamp until later and just keep adding tournaments for the time being. This is impracticable for technical reasons (i.e. I broke something on the backend and I don't know how to fix it, and also because the current way I'm storing files is primitive and not forward-compatible with the new version of the site that I am working on, so I would have to redo all the data entry when I revamp the site, which is something I do not want to do).

It is also something I do not want to do because I have come to agree with various arguments put forth (in particular by Gordon Arsenoff) that the way I rank teams is inadequate and quite honestly misleading. There are distinct advantages and disadvantages to doing things like including college tournaments in the rankings and weighting sets based on difficulty and so forth. To solve this problem, I would like to create an ensemble of rankings using various parameters (e.g. college tournaments included or not; sets weighted based on difficulty or on perceived quality or not; different ways of handling teams lacking a full complement [e.g. reduce game weightage; remove games altogether; do something more complicated; or something else altogether]; etc.) from which quizbowl commentators more educated than me could select the ranking that best reflects their view of the parameters that come together to make a team good. Unfortunately, this is also technically impracticable at this time. As such, it is my view that these rankings will be most effective once I am able to fully rewrite their code to make these kinds of things feasible.

I will try to have everything ready by the beginning of May or so, so that these rankings can be of use for nationals prognostication. That failing, I will certainly have a polished version ready for the beginning of the next competition year.

P.S. Nobody besides me is working on these rankings. Particularly not other members of Yale's quizbowl club. And most particularly not anyone whose name might sound like Batt Hackson. No good has ever come of inquiring about these rankings with people whose name sounds like Batt Hackson.

Thanks for your work on this. I've thought about trying to do a rating system of my own before and have even started a small one in excel. The most difficult part seems to be school identification. It would be most helpful if we had as a starting point a list of all high schools with some form of ID number that could then be reported with statistics from tournaments. The American Mathematics Competition asks each school for their CEEB #, which is an already existing system with an ID number for each high school in the US. I don't know if we would have to get permission from the College Board to use these numbers or not. If it was OK to use them it would be very helpful for rating teams to get that ID number included as a column in SQBS data output. We could add an additional digit to code for A, B, or C team and we would be ready to go. Best of luck with your rewrite, and know that your efforts have been appreciated.

You have indeed stumbled across the most tedious part of data entry for this (and the part which makes this most difficult to automate). Rather than using the CEEB code, my current plan for identification numbers is to use the org_id that NAQT assigns to all schools (e.g. South Jefferson Senior High School is 60688) and then append some sort of team-letter-related suffix. The main advantage this provides is that I don't have to go and manually locate NAQT org_id's when I'm scraping results data from NAQT's website, because this information is already on their website. As such, there's no need to translate these ID numbers into a new numbering system (as I was doing before using my own dumb numbering system, or as I would have to do if I were to use CEEB codes).

Though CEEB codes may be more complete in terms of listing all schools in existence than NAQT might be, it seems like NAQT makes a point of adding codes for schools that participate in anything that resembles quizbowl, so I don't think completeness of the numbering scheme will be an issue with NAQT.

One potential issue (which I would have to ask Jeff or someone about) is that if NAQT decides to change their numbering system altogether in any way at all (including shuffling schools from one number to another and so forth), this would be a bit of a hassle for me. If NAQT has no plans to change their numbering system in the foreseeable future (or if I can at least get a heads up in advance if the system is going to change), I will probably implement a system based on NAQT's org_id's.

Excelsior (smack) wrote:One potential issue (which I would have to ask Jeff or someone about) is that if NAQT decides to change their numbering system altogether in any way at all (including shuffling schools from one number to another and so forth), this would be a bit of a hassle for me. If NAQT has no plans to change their numbering system in the foreseeable future (or if I can at least get a heads up in advance if the system is going to change), I will probably implement a system based on NAQT's org_id's.

I don't speak officially for NAQT, but I probably know at least as much about database things as Jeff does, so I'll speak unofficially. We don't anticipate changing that column in the foreseeable future, and if we did, it's not like those are a trade secret—we could probably arrange to get you the necessary data to switch things over.

Do schools know their own NAQT ID numbers? Could this be a line added to registration forms? Is it too much to ask tournament directors to look up ID numbers and include them in their statistical output?

jonah wrote:I don't speak officially for NAQT, but I probably know at least as much about database things as Jeff does, so I'll speak unofficially. We don't anticipate changing that column in the foreseeable future, and if we did, it's not like those are a trade secret—we could probably arrange to get you the necessary data to switch things over.

Sounds good. Thanks, Jonah.

Do schools know their own NAQT ID numbers? Could this be a line added to registration forms? Is it too much to ask tournament directors to look up ID numbers and include them in their statistical output?

I find it rather unlikely that schools would know their own NAQT ID numbers, though it's of course not difficult to look up. To include it in statistical output, however, would either require a new version of SQBS or a program to post-process SQBS output and incorporate ID numbers. I don't know if the developer of SQBS would have any interest in doing the former, and I'm not sure that it would be reasonable to expect TDs to do the latter. Maybe in the future, when there is more use for a unified school-numbering system for quizbowl, reporting school ID numbers could become standard practice, but for the time being, these rankings are the only real use I can see for that information.

HSQBRank, what is best in life?

To crush your enemy ranking systems, see them driven before you, and to hear the lamentation of their programmers.

I'll confirm Jonah's note that we have no plans to change our ID system.

Out of curiosity, does it look like these will be ready in May or is it looking more like a 2012-2013 restart?

Thanks!

mithokie wrote:Out of curiosity, does it look like these will be ready in May or is it looking more like a 2012-2013 restart?

Thanks!

Ah, sorry for the late response. At this point, you probably won't be surprised to learn that I'm not going to restart this until the 2012-13 school year.