Excelsior Rankings (dead forever)

Dormant threads from the high school sections are preserved here.
User avatar
theflyingdeutschman
Wakka
Posts: 101
Joined: Sat Mar 20, 2010 8:52 pm

Re: Excelsior Rankings - now live for 2011-12

Post by theflyingdeutschman »

Fred wrote:Saying that "most" housewrites have a roughly equivalent difficulty is, well... wrong. Sorry, but HFT had definite differences from BHSAT and LIST.
But those housewrites are more similar to HSAPQ than they are to NAQT in my opinion.
Sam Deutsch
Richard Montgomery '13
University Of Southern California '17
User avatar
Whiter Hydra
Auron
Posts: 1418
Joined: Tue Dec 04, 2007 8:46 pm
Location: Fairfax, VA
Contact:

Re: Excelsior Rankings - now live for 2011-12

Post by Whiter Hydra »

theflyingdeutschman wrote:
Fred wrote:Saying that "most" housewrites have a roughly equivalent difficulty is, well... wrong. Sorry, but HFT had definite differences from BHSAT and LIST.
But those housewrites are more similar to HSAPQ than they are to NAQT in my opinion.
In terms of format, yes. In terms of difficulty, that's a sticker question.
Harry White
TJHSST '09, Virginia Tech '13

Owner of Tournament Database Search and Quizbowl Schedule Generator
Will run stats for food
Joshua Rutsky
Tidus
Posts: 663
Joined: Mon Oct 24, 2005 9:59 am
Location: Hoover, AL

Re: Excelsior Rankings - now live for 2011-12

Post by Joshua Rutsky »

I'm sure this is obvious, but it will need to be dealt with for this to continue to be helpful. Should we notify you about squads and their composition when there are JV brackets at an event? Hoover A, for example, is a varsity team, but Hoover B is our JV squad, and would be Hoover JV A at any other event. Hoover B is another varsity squad that didn't attend for various reasons. Is there a manner that TDs can designate this to make it helpful for your work, or should we notify you on a case-by-case basis?
Joshua Rutsky
VP for Curriculum and Camp Operations, Qwiz
ASCA Board Member
Hoover High School Coach (Retired)
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings - now live for 2011-12

Post by Excelsior (smack) »

Even if I were to implement a system by means of which TDs/coaches/players could indicate the appropriate letter designations of teams, I am doubtful that it would be particularly useful: A lot of information about this sort of thing is already posted on the boards, often to help Fred with his rankings, and I don't think that adding an additional facility by means of which to report this information specifically to me would get very many other people to post this information. I do read the forums regularly, and will act on any relevant information that is posted here. I will probably also continue to draw on information about team strength found in Fred's rankings if he continues to include that information.

As always, if you are browsing through the website and find something amiss, you can click on the "Submit correction/information/error report" found at the bottom of every page and send me information if you don't want to clutter up the forums or whatever. I will act on the information sooner and/or later.

I have made the modification you requested, though it will not be reflected in the rankings until I input stats for another tournament.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
Edward Powers
Auron
Posts: 1107
Joined: Thu Sep 25, 2008 2:52 pm

Re: Excelsior Rankings (updated Oct. 5, 2011)

Post by Edward Powers »

Ashvin,

I noticed one error, an understandable one, on your October 5th update.

You accidentally added the PHSAT results for Blair Academy of New Jersey to the Maryland Fall Tournament results of Montgomery Blair High School of Maryland. The error probably occurred because Montgomery Blair is often referred to simply as "Blair", and was so designated at Maryland's Fall Tournament. Still, these records should be disentangled, restoring Montgomery Blair to its earlier 8-4 record while giving Blair Academy of NJ its own separate record.
Ed Powers
Coach
SJHS Academic Team
Metuchen, NJ
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Oct. 5, 2011)

Post by Excelsior (smack) »

Edward Powers wrote:Ashvin,

I noticed one error, an understandable one, on your October 5th update.

You accidentally added the PHSAT results for Blair Academy of New Jersey to the Maryland Fall Tournament results of Montgomery Blair High School of Maryland. The error probably occurred because Montgomery Blair is often referred to simply as "Blair", and was so designated at Maryland's Fall Tournament. Still, these records should be disentangled, restoring Montgomery Blair to its earlier 8-4 record while giving Blair Academy of NJ its own separate record.
Fixed.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
Charles Martel
Wakka
Posts: 199
Joined: Sun Jan 24, 2010 1:21 am

Re: Excelsior Rankings (updated Oct. 20, 2011)

Post by Charles Martel »

Why haven't teams that played at UIUC Earlybird been added to the rankings yet?
Adam Kalinich
MIT 2012-
Illinois Math and Science Academy 2009-2012
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Oct. 20, 2011)

Post by Excelsior (smack) »

whitesoxfan wrote:Why haven't teams that played at UIUC Earlybird been added to the rankings yet?
I forgot to mark the Earlybird data as ready for addition to the rankings. Rectified.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
Charles Martel
Wakka
Posts: 199
Joined: Sun Jan 24, 2010 1:21 am

Re: Excelsior Rankings (updated Oct. 20, 2011)

Post by Charles Martel »

Tristan Wiley = Macomb's data doesn't appear to be entered.
Adam Kalinich
MIT 2012-
Illinois Math and Science Academy 2009-2012
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Oct. 20, 2011)

Post by Excelsior (smack) »

whitesoxfan wrote:Tristan Wiley = Macomb's data doesn't appear to be entered.
This is true. I had originally excised all of Tristan's matches because I assumed there were some peculiar circumstances surrounding him being Tristan rather than Macomb, but I guess he must've just been playing unaffiliated or something. They will be re-incorporated in the next update.

While I'm here, I should note that as of now, the rankings are not terribly useful. This is because there are too many disjoint circuits that have not yet had any interaction with each other. Luckily, Harvard Fall seems to be drawing teams from a sufficiently ridiculous number of places that every major circuit east of the Rockies will be unified after HFT happens. Starting then, the rankings will become increasingly relevant.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Oct. 27, 2011)

Post by Excelsior (smack) »

All MOO and MAGNI sites that have occurred so far have been incorporated into the rankings (along with the Florida Gateway Invitational, a college tournament run on an A-set). This has a few consequences.

First, the rankings are no longer continuously numbered. For example, as of this writing, Ladue A is ranked 4th, while the next team, GDS A, is ranked 6th. This is because there is a college team (in this case, Michigan A) ranked 5th. College teams are of course not displayed in the rankings list, but games played against college teams do affect the rankings.

Second, for those of you who are amused by this sort of curiosity, you may now view the rankings of colleges for which data is currently available. I trust interested college players to put very little stock in the rankings, as the circuit is not yet connected; college teams have even higher lineup variability than high school teams; I didn't really bother to account for weakened house teams; etc.

Also: you can now see a snapshot of the state of connectedness of the circuit. It will be updated every so often, but not necessarily at the same time as the rankings (for technical reasons). As you can see, there are (as of this writing) 23 disjoint circuits. I recognize that the page is not exactly user-friendly (it displays team ID numbers, not team names); I may fix this when I get some time over Thanksgiving.
Last edited by Excelsior (smack) on Thu Oct 27, 2011 11:27 pm, edited 1 time in total.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
Adventure Temple Trail
Auron
Posts: 2754
Joined: Tue Jul 15, 2008 9:52 pm

Re: Excelsior Rankings (updated Oct. 27, 2011)

Post by Adventure Temple Trail »

Is there a way to paper over the non-continuous rankings numbers with continuous rankings numbers? If this ends up being something that high school teams might be able to point to in order to, say, secure funding for Nationals, it's probably better for them to be able to say "We're 25th on this statistical rankings page" than "Well, uh, the number says 58, but there are these invisible teams that I may have miscounted..."

(I don't know anything about programming, but) It seems like it should be possible to have a hidden list of rankings numbers for HS and college together, but output an "HSRanking" column on the high school page which attributes each team a number equal to (number of high school teams ranked above this team + 1). Perhaps the same thing could be done for the college page.

I eagerly await the effects of increased cross-circuit travel on these rankings - they could get interesting and relevant quick.
Matt Jackson
University of Chicago '24
Yale '14, Georgetown Day School '10
member emeritus, ACF
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Oct. 27, 2011)

Post by Excelsior (smack) »

I have now updated the to-do list with the list of things I plan to do. Your suggestion regarding renumbering the rankings is on this list.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
Charbroil
Auron
Posts: 1146
Joined: Fri Jun 09, 2006 11:52 am
Location: St. Charles, MO

Re: Excelsior Rankings (updated Oct. 27, 2011)

Post by Charbroil »

Are you not going to incorporate ACF Novice into these rankings?
Charles Hang
Francis Howell Central '09
St. Charles Community College '14
Washington University in St. Louis '19, 2x (President, 2017-19)

Owner, Olympia Academic Competition Questions, LLC
Question Writer, National Academic Quiz Tournaments, LLC and National History Bee and Bowl
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Oct. 27, 2011)

Post by Excelsior (smack) »

Charbroil wrote:Are you not going to incorporate ACF Novice into these rankings?
No. I have literally no way of correlating teams at Collegiate Novice with the teams that will be playing regular college tournaments throughout the year. While I'm usually a fan of having more data, Collegiate Novice data would be so actively unhelpful that I think they would reduce the quality of the rankings.

In a similar vein, I generally do not include novice (or otherwise restricted-eligibility) high school tournaments in the rankings. (I do tend to include JV divisions of tournaments that also have a Varsity division, though, because in those cases it is generally fairly reasonable to assume that if there are e.g. two teams from a school in the varsity division and two in the JV division, that the latter two can be considered "C" and "D" teams.)

(This is also a separate issue from if I will include high school tournaments run on Collegiate Novice in the rankings; if any such tournaments exist, I will.)
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Nov. 10, 2011)

Post by Excelsior (smack) »

And now that Harvard Fall has been added to these rankings, I present you with a curated list of the top high school teams in the nation, according to the rankings. Teams have been removed from the list in cases where they are not linked in to the national circuit that now unifies most of the country.

Fred's ranking of each team (as of 13 Nov 2011 and as of 17 Nov 2011) is given in brackets. Teams with asterisks next to their names have especially unreliable rankings, because they have no losses yet.

1. [3|2] Centennial*
2. [6|10] duPont Manual*
3. [16|23] University School of Nashville*
4. [18|20] Ladue*
5. [20|42] Early College at Guilford
6. [4|4] Rockford Auburn
7. [1|1] Hunter
8. [38|40] Hawken*
9. [40|19] Brookwood
10. [9|11] IMSA
11. [7|7] Dorman
12. [11|8] DCC
13. [13|5] Thomas Jefferson
14. [14|14] Dunbar A
15. [46|48] Northmont
16. [10|12] Loyola
17. [61|65] Dunbar B
18. [12|13] St. Anselm's
19. [98|UR] Walnut Hills
20. [UR|79] Alpharetta
21. [88|93] Pope John Paul II
22. [8|6] Seven Lakes
23. [36|38] Warren G. Harding
24. [63|58] Ezell-Harding
25. [45|47] East Chapel Hill

High-ranked teams that are not linked into the national circuit:
[UR|UR] Detroit Country Day [undefeated at HFT Michigan]
[UR|77] West Des Moines Valley [undefeated at Matt Cvijanovich Memorial]
[25|31] Cistercian B [undefeated at TQBA Kickoff]
[UR|UR] Edmond Memorial B [undefeated at OQBA Tulsa Fall]
[UR|UR] Dunbar D [undefeated at the JV division of Danville Fall]
[UR|UR] William Henry Harrison [undefeated at North White Kickoff]
[60|64] Wayzata [undefeated at Olaf Learnament]



At this time, California (separated into North and South), Minnesota, parts of Texas, Indiana, Oklahoma, parts of Michigan, Iowa, and southern Florida are not connected to the rest of the circuit. I expect all of these besides California and maybe southern Florida (depending on whether or not Ransom Everglades travels, basically) to become linked into the main circuit by March or so.

EDIT: Fred Nov 17 rankings have also been added as the second number in brackets.
Last edited by Excelsior (smack) on Thu Nov 17, 2011 8:21 pm, edited 1 time in total.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
dtaylor4
Auron
Posts: 3733
Joined: Tue Nov 16, 2004 11:43 am

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by dtaylor4 »

The best avenue Iowa will have to get linked to the national circuit is through Bettendorf, which will be at the same tournament as Belvidere North this weekend.

EDIT: The only reasonable chance that comes to mind is at NAQT State, which will probably be a disaster again.
User avatar
Remember-the-Alamo-Remember-Goliad
Wakka
Posts: 151
Joined: Thu May 27, 2010 10:28 am

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Remember-the-Alamo-Remember-Goliad »

Here are the results of the Edmond Memorial Bulldog Tournament run on IS-109 on Nov. 19, 2011

http://naqt.com/stats/tournament-teams. ... nt_id=3386
Gregory Schweers
County of Dallas
City of Irving
Cistercian Preparatory School Team
The only Monk on this whole board
User avatar
The Friar
Wakka
Posts: 159
Joined: Fri Jul 10, 2009 2:39 pm

Re: Excelsior Rankings (updated Oct. 27, 2011)

Post by The Friar »

Excelsior (smack) wrote:
Charbroil wrote:Are you not going to incorporate ACF Novice into these rankings?
No. I have literally no way of correlating teams at Collegiate Novice with the teams that will be playing regular college tournaments throughout the year. While I'm usually a fan of having more data, Collegiate Novice data would be so actively unhelpful that I think they would reduce the quality of the rankings.
I don't think including these data would be actively unhelpful with regard to high school teams who play ACF Novice, as long as college teams at ACF Novice are simply treated as different entities from the teams with the same names playing unrestricted events (i.e., define teams such that Wash U A can't play ACF Novice, but Wash U Novice A can). With that treatment, data from restricted-eligibility college events would be no worse than a small amount of noise.

On the other hand, I've got a problem with including any college tournaments in your high school team rankings, which I'll discuss in my next post.
Gordon Arsenoff
Rochester '06
WUStL '14 (really)

Developer of WUStL Updates Statistics Live!
User avatar
The Friar
Wakka
Posts: 159
Joined: Fri Jul 10, 2009 2:39 pm

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by The Friar »

There are two really big issues with the way college tournaments are incorporated here.

First, high school teams appear to suffer a ranking penalty for playing college tournaments, if games are weighted by the difficulty of the set they were played on. It's been noted already that college tournaments are overweighted by the current scheme, because they are less similar to nationals than nationals would be but they get higher weight than nationals would. With college tournaments included, however, set difficulty is positively correlated with opponent strength -- college teams are, in principle, the strongest opponents, and they are only faced at the toughest tournaments. Since weight is monotonically increasing in set difficulty, and probability of losing in opponent strength, it follows that weight is positively correlated with probability of losing for high school teams who play college tournaments. A few Ls against college opposition much stronger than you, on questions that don't look much like nationals, lower your ranking many times more than your ranking is raised by a few Ws, on questions that don't look much like nationals, against high school opposition much weaker than you -- even though the latter are theoretically potential nationals opponents and the former are not. In fact, this is probably true to some extent for high school tournaments as well -- better teams on average seek out harder tournaments -- but the effect size should be small; with the inclusion of college tournaments, however, the effect is stark.

The second big issue is one of security. Eligibility rules in some states frown on high school teams playing college tournaments, and those teams who do so anyway often play under fictitious names to avoid repercussions with state bureaucracies. The rankings do not appear to respect this anonymity, but instead go ahead and translate those fictitious names into real ones, eliminating teams' anonymity and leaving them open to eligibility recriminations.

The motivating example for both these concerns is the performance of Rock Bridge A at Missiles of October, where they took several losses to college teams under the original team name of "Roseanne Barr". Rock Bridge A appears vastly underrated by Excelsior in comparison to the top 25 poll or Fred Morlan's top 100, while other top (Missouri) teams are ranked fairly similarly in all these publications, and the driver of Rock Bridge's low ranking seems to be the losses they took against college teams (there are almost no other losses).
Gordon Arsenoff
Rochester '06
WUStL '14 (really)

Developer of WUStL Updates Statistics Live!
User avatar
The Friar
Wakka
Posts: 159
Joined: Fri Jul 10, 2009 2:39 pm

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by The Friar »

Ultimately, I recommend replacing the KRACH probability model at the heart of these rankings with DP-FRIAR in order to both: (a) incorporate all the information contained in scores as opposed to just W-L records; and (b) estimate difficulty from the data, at the packet level, instead of arbitrarily assuming difficulty levels for each set.

EDIT: using, of course, the more canonical logit parametrization for p as found in this post rather than the version given in the initial post.
Gordon Arsenoff
Rochester '06
WUStL '14 (really)

Developer of WUStL Updates Statistics Live!
User avatar
tinioril
Lulu
Posts: 61
Joined: Sun Nov 20, 2011 10:36 am
Location: Chicago

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by tinioril »

Update with IHSSBCA kickoffs?
Ned Lauber
Wheaton Warrenville South '12
Vanderbilt '16
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Oct. 27, 2011)

Post by Excelsior (smack) »

First, high school teams appear to suffer a ranking penalty for playing college tournaments, if games are weighted by the difficulty of the set they were played on. It's been noted already that college tournaments are overweighted by the current scheme, because they are less similar to nationals than nationals would be but they get higher weight than nationals would. With college tournaments included, however, set difficulty is positively correlated with opponent strength -- college teams are, in principle, the strongest opponents, and they are only faced at the toughest tournaments. Since weight is monotonically increasing in set difficulty, and probability of losing in opponent strength, it follows that weight is positively correlated with probability of losing for high school teams who play college tournaments. A few Ls against college opposition much stronger than you, on questions that don't look much like nationals, lower your ranking many times more than your ranking is raised by a few Ws, on questions that don't look much like nationals, against high school opposition much weaker than you -- even though the latter are theoretically potential nationals opponents and the former are not. In fact, this is probably true to some extent for high school tournaments as well -- better teams on average seek out harder tournaments -- but the effect size should be small; with the inclusion of college tournaments, however, the effect is stark.
This is true. The existing implementation of incorporating college tournaments is inadequate, and I am planning on fixing that. My proposed solution: construct two sets of rankings - a "high school ranking" and a "college ranking". Both rankings will use all the data available to me, but will weight them differently.

The high school ranking will maximally weight tournaments that are exclusive to high school teams and are of approximately HSNCT/NSC difficulty. The more a tournament deviates in difficulty from HSNCT/NSC in either direction, the lower the weight given to that tournament. In addition, tournaments that are non-exclusive to high school teams will have their weight reduced to accommodate for the increased strength of competition that one might face there.

On the other hand, the college ranking will maximally weight tournaments that are of approximately DI ICT/ACF Nats difficulty. It seems a fair assumption that these are the most difficult tournaments of the year, excluding summer opens (which are of no interest to me); as such, easier tournaments will have lower weights. I think it may be appropriate to add a weight-reduction to tournaments at which both open and non-open teams play (e.g. MO this year), but I'm not sure about that.

How does that sound? If it turns out that the incorporation of tournaments with college teams damages the rankings to the point that it would be better to do without them, I can certainly do away with them. (High school teams are necessary to link the college circuit pre-ICT/ACF Nats, but not vice versa, except insofar as California links into the circuit sooner if college teams are included [via ICT])
The second big issue is one of security. Eligibility rules in some states frown on high school teams playing college tournaments, and those teams who do so anyway often play under fictitious names to avoid repercussions with state bureaucracies. The rankings do not appear to respect this anonymity, but instead go ahead and translate those fictitious names into real ones, eliminating teams' anonymity and leaving them open to eligibility recriminations.
I was under the impression that playing pseudonymously was a mere formality for the purposes of not technically being affiliated with one's school and that nobody was actually fooled by them. If this is not the case, I will certainly stop doing that.
Ultimately, I recommend replacing the KRACH probability model at the heart of these rankings with DP-FRIAR in order to both: (a) incorporate all the information contained in scores as opposed to just W-L records;
I don't think I actually understand DP-FRIAR. Nonetheless, let me try: We obtain a pair of binomial distributions for Y (total score) and Y_1|Y (a team's score given a particular total score). We sample multiple data points for Y by examining all games played on a particular packet. This much makes sense. How do we then sample multiple data points for Y_1|Y?

In any case, once we've sampled multiple points for Y and Y_1|Y, we can then estimate p based on the distribution for Y; what is the analogous parameter for the Y_1|Y distribution? Is it still ((p_1 - p_2) / 2(p_1*p_2)) as given in your first post, or is it something else?

So once we've determined p and whatever the parameter is for the Y_1|Y distribution, we use the "canonical logit parametrization" for p you gave in the second post to compute r_1 and r_2. These r_1 and r_2 are the ultimate output of the process that I would then use to rank teams.

In order to compare different tournaments, we would then need to use the modified distributions Y | different_set with some value of p_D (which we also determine from the data?).
(b) estimate difficulty from the data, at the packet level, instead of arbitrarily assuming difficulty levels for each set.
How does DP-FRIAR behave without access to packet-level data (that is, we don't know which games occurred on which packets)? Also, does it even work if I don't have access to scores (but instead only win-loss records)?

Do note that I was going to start determining set-level difficulty from PPB data eventually; the arbitrary weight's I've picked so far are just stopgaps for the meantime.



EDIT:
Update with IHSSBCA kickoffs?
Probably not until past New Year's.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
jonpin
Auron
Posts: 2266
Joined: Wed Feb 04, 2004 6:45 pm
Location: BCA NJ / WUSTL MO / Hackensack NJ

Re: Excelsior Rankings (updated Oct. 27, 2011)

Post by jonpin »

Excelsior (smack) wrote:Do note that I was going to start determining set-level difficulty from PPB data eventually; the arbitrary weight's I've picked so far are just stopgaps for the meantime.
I might have gotten an answer for this previously, but I don't recall it... I still don't understand what the motivation is for "harder tournament = more important".
Jon Pinyan
Coach, Bergen County Academies (NJ); former player for BCA (2000-03) and WUSTL (2003-07)
HSQB forum mod, PACE member
Stat director for: NSC '13-'15, '17; ACF '14, '17, '19; NHBB '13-'15; NASAT '11

"A [...] wizard who controls the weather" - Jerry Vinokurov
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Oct. 27, 2011)

Post by Excelsior (smack) »

jonpin wrote:
Excelsior (smack) wrote:Do note that I was going to start determining set-level difficulty from PPB data eventually; the arbitrary weight's I've picked so far are just stopgaps for the meantime.
I might have gotten an answer for this previously, but I don't recall it... I still don't understand what the motivation is for "harder tournament = more important".
In theory, once I modify the weightings as mentioned in my previous post (to devalue college tournaments for ranking high school teams), these rankings will in some sense be an approximation of how teams will perform at nationals. We expect performance at nationals to correlate most closely with sets that are most similar to the sets that will be used at nationals. The only quantifiable measure of similarness of sets (that I can think of, anyway) is their difficulty; as such, it seems to me that weighting sets that are of comparable difficulty to HSNCT/NSC the most heavily would yield the most useful rankings.

That said, given various discussions surrounding HFT this year, I am not sure if my position is correct. It is certainly true that there exists some difficulty level below which a better team will not win against a worse team nearly as often as it would on HSNCT/NSC (and consequently, games played on sets below that difficulty level ought to have reduced weight). Empirically speaking, however, I don't know what that difficulty level is. It may well be that everything harder than A-sets have similar odds of victory for national-contending teams as do nationals-level sets; or it might be that even A-sets have similar-enough odds; or maybe we need things harder than IS-sets to get close-enough odds.

(Whatever the case may be, my weights are too extreme right now, I'm pretty sure, even ignoring the college sets that are currently included; I think that a spread of ~0.85 to 1.2 from A-sets to HSNCT/NSC would adequately capture the variation in victory odds across those sets, though this estimate is based only on intuition and not on any actual rigorous analysis of data.)
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
Kahloon
Lulu
Posts: 79
Joined: Mon Jun 01, 2009 9:37 am

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Kahloon »

This doesn't look this has been added: (HFT mirror at WKU)
viewtopic.php?f=1&t=11551&p=230002#p230002

Where Lexington A = Dunbar
Idrees Kahloon
Dunbar '12
User avatar
Charles Martel
Wakka
Posts: 199
Joined: Sun Jan 24, 2010 1:21 am

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Charles Martel »

He's only added tournaments from November 12th and earlier right now.
Adam Kalinich
MIT 2012-
Illinois Math and Science Academy 2009-2012
User avatar
Down and out in Quintana Roo
Auron
Posts: 2907
Joined: Wed Apr 09, 2008 7:25 am
Location: Camden, DE
Contact:

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Down and out in Quintana Roo »

It would be nice to see an update to the rankings, since there are a lot of tournaments yet to be entered. I'm sure there will be some drastic changes in the numbers. It'll also help people quickly access stats for teams if they're thinking about voting in the Mid-Season Poll that George is running.

This is still really cool to finally have a comprehensive one of these.
Mr. Andrew Chrzanowski
Caesar Rodney High School
Camden, Delaware
CRHS '97-'01
University of Delaware '01-'05
CRHS quizbowl coach '06-'12
http://crquizbowl.edublogs.org
User avatar
Adventure Temple Trail
Auron
Posts: 2754
Joined: Tue Jul 15, 2008 9:52 pm

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Adventure Temple Trail »

large, visible red letters on the Excelsior web page wrote:The rankings may not be updated until after New Year's, because I am currently in the midst of rewriting the entire site to make it more maintainable and extensible.
They're large, visible, and red!
Matt Jackson
University of Chicago '24
Yale '14, Georgetown Day School '10
member emeritus, ACF
User avatar
Charles Martel
Wakka
Posts: 199
Joined: Sun Jan 24, 2010 1:21 am

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Charles Martel »

It's after New Year's. I agree with Mr. Chrzanowski that these rankings are great, and it would be useful if we could get an update to help us vote in the Mid-Season Poll.
Adam Kalinich
MIT 2012-
Illinois Math and Science Academy 2009-2012
User avatar
AKKOLADE
Sin
Posts: 15783
Joined: Thu Apr 24, 2003 8:08 am

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by AKKOLADE »

Ashvin I will Excelsior (smack) you if you don't update your rankings right now.

Edit: It's been two whole days since the New Year YOU MONSTER
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
Down and out in Quintana Roo
Auron
Posts: 2907
Joined: Wed Apr 09, 2008 7:25 am
Location: Camden, DE
Contact:

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Down and out in Quintana Roo »

Really i just asked because of George's poll. Relax.
Mr. Andrew Chrzanowski
Caesar Rodney High School
Camden, Delaware
CRHS '97-'01
University of Delaware '01-'05
CRHS quizbowl coach '06-'12
http://crquizbowl.edublogs.org
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Excelsior (smack) »

Owing to multiple factors (most of which pertain to me being lazy, bad at coding, and also delinquent on other quizbowl-related duties), the Excelsior Rankings will not be updated in the near future. I have received some suggestions that I put off the site revamp until later and just keep adding tournaments for the time being. This is impracticable for technical reasons (i.e. I broke something on the backend and I don't know how to fix it, and also because the current way I'm storing files is primitive and not forward-compatible with the new version of the site that I am working on, so I would have to redo all the data entry when I revamp the site, which is something I do not want to do).

It is also something I do not want to do because I have come to agree with various arguments put forth (in particular by Gordon Arsenoff) that the way I rank teams is inadequate and quite honestly misleading. There are distinct advantages and disadvantages to doing things like including college tournaments in the rankings and weighting sets based on difficulty and so forth. To solve this problem, I would like to create an ensemble of rankings using various parameters (e.g. college tournaments included or not; sets weighted based on difficulty or on perceived quality or not; different ways of handling teams lacking a full complement [e.g. reduce game weightage; remove games altogether; do something more complicated; or something else altogether]; etc.) from which quizbowl commentators more educated than me could select the ranking that best reflects their view of the parameters that come together to make a team good. Unfortunately, this is also technically impracticable at this time. As such, it is my view that these rankings will be most effective once I am able to fully rewrite their code to make these kinds of things feasible.

I will try to have everything ready by the beginning of May or so, so that these rankings can be of use for nationals prognostication. That failing, I will certainly have a polished version ready for the beginning of the next competition year.

P.S. Nobody besides me is working on these rankings. Particularly not other members of Yale's quizbowl club. And most particularly not anyone whose name might sound like Batt Hackson. No good has ever come of inquiring about these rankings with people whose name sounds like Batt Hackson.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
mithokie
Rikku
Posts: 345
Joined: Mon Apr 17, 2006 2:39 pm
Location: Blacksburg, VA

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by mithokie »

Thanks for your work on this. I've thought about trying to do a rating system of my own before and have even started a small one in excel. The most difficult part seems to be school identification. It would be most helpful if we had as a starting point a list of all high schools with some form of ID number that could then be reported with statistics from tournaments. The American Mathematics Competition asks each school for their CEEB #, which is an already existing system with an ID number for each high school in the US. I don't know if we would have to get permission from the College Board to use these numbers or not. If it was OK to use them it would be very helpful for rating teams to get that ID number included as a column in SQBS data output. We could add an additional digit to code for A, B, or C team and we would be ready to go. Best of luck with your rewrite, and know that your efforts have been appreciated.
Matt Beeken
Eastern Montgomery High School
Math Instructional Coach
Scholastic Bowl Coach (2022 - ??)
mbeeken AT mcps DOT org
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Excelsior (smack) »

You have indeed stumbled across the most tedious part of data entry for this (and the part which makes this most difficult to automate). Rather than using the CEEB code, my current plan for identification numbers is to use the org_id that NAQT assigns to all schools (e.g. South Jefferson Senior High School is 60688) and then append some sort of team-letter-related suffix. The main advantage this provides is that I don't have to go and manually locate NAQT org_id's when I'm scraping results data from NAQT's website, because this information is already on their website. As such, there's no need to translate these ID numbers into a new numbering system (as I was doing before using my own dumb numbering system, or as I would have to do if I were to use CEEB codes).

Though CEEB codes may be more complete in terms of listing all schools in existence than NAQT might be, it seems like NAQT makes a point of adding codes for schools that participate in anything that resembles quizbowl, so I don't think completeness of the numbering scheme will be an issue with NAQT.

One potential issue (which I would have to ask Jeff or someone about) is that if NAQT decides to change their numbering system altogether in any way at all (including shuffling schools from one number to another and so forth), this would be a bit of a hassle for me. If NAQT has no plans to change their numbering system in the foreseeable future (or if I can at least get a heads up in advance if the system is going to change), I will probably implement a system based on NAQT's org_id's.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
jonah
Auron
Posts: 2383
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by jonah »

Excelsior (smack) wrote:One potential issue (which I would have to ask Jeff or someone about) is that if NAQT decides to change their numbering system altogether in any way at all (including shuffling schools from one number to another and so forth), this would be a bit of a hassle for me. If NAQT has no plans to change their numbering system in the foreseeable future (or if I can at least get a heads up in advance if the system is going to change), I will probably implement a system based on NAQT's org_id's.
I don't speak officially for NAQT, but I probably know at least as much about database things as Jeff does, so I'll speak unofficially. We don't anticipate changing that column in the foreseeable future, and if we did, it's not like those are a trade secret—we could probably arrange to get you the necessary data to switch things over.
Jonah Greenthal
National Academic Quiz Tournaments
mithokie
Rikku
Posts: 345
Joined: Mon Apr 17, 2006 2:39 pm
Location: Blacksburg, VA

Re: Excelsior Rankings (temporarily inactive until at least May)

Post by mithokie »

Do schools know their own NAQT ID numbers? Could this be a line added to registration forms? Is it too much to ask tournament directors to look up ID numbers and include them in their statistical output?
Matt Beeken
Eastern Montgomery High School
Math Instructional Coach
Scholastic Bowl Coach (2022 - ??)
mbeeken AT mcps DOT org
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (updated Nov. 16, 2011)

Post by Excelsior (smack) »

jonah wrote:I don't speak officially for NAQT, but I probably know at least as much about database things as Jeff does, so I'll speak unofficially. We don't anticipate changing that column in the foreseeable future, and if we did, it's not like those are a trade secret—we could probably arrange to get you the necessary data to switch things over.
Sounds good. Thanks, Jonah.
Do schools know their own NAQT ID numbers? Could this be a line added to registration forms? Is it too much to ask tournament directors to look up ID numbers and include them in their statistical output?
I find it rather unlikely that schools would know their own NAQT ID numbers, though it's of course not difficult to look up. To include it in statistical output, however, would either require a new version of SQBS or a program to post-process SQBS output and incorporate ID numbers. I don't know if the developer of SQBS would have any interest in doing the former, and I'm not sure that it would be reasonable to expect TDs to do the latter. Maybe in the future, when there is more use for a unified school-numbering system for quizbowl, reporting school ID numbers could become standard practice, but for the time being, these rankings are the only real use I can see for that information.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
AKKOLADE
Sin
Posts: 15783
Joined: Thu Apr 24, 2003 8:08 am

Re: Excelsior Rankings (temporarily inactive until at least May)

Post by AKKOLADE »

HSQBRank, what is best in life?

To crush your enemy ranking systems, see them driven before you, and to hear the lamentation of their programmers.
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6113
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: Excelsior Rankings (temporarily inactive until at least May)

Post by Important Bird Area »

I'll confirm Jonah's note that we have no plans to change our ID system.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
mithokie
Rikku
Posts: 345
Joined: Mon Apr 17, 2006 2:39 pm
Location: Blacksburg, VA

Re: Excelsior Rankings (temporarily inactive until at least May)

Post by mithokie »

Out of curiosity, does it look like these will be ready in May or is it looking more like a 2012-2013 restart?

Thanks!
Matt Beeken
Eastern Montgomery High School
Math Instructional Coach
Scholastic Bowl Coach (2022 - ??)
mbeeken AT mcps DOT org
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings (temporarily inactive until at least May)

Post by Excelsior (smack) »

mithokie wrote:Out of curiosity, does it look like these will be ready in May or is it looking more like a 2012-2013 restart?

Thanks!
Ah, sorry for the late response. At this point, you probably won't be surprised to learn that I'm not going to restart this until the 2012-13 school year.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
Locked