A quick-and-dirty player ranking stat (I'm not first)

[theMoMA]

Is it gauche to quote oneself?

It's folly to use something like "power percentage" or "domination of a particular category" or "subjective assessment of 'good buzzes'" to assess whether a player can consistently get tossups. Rather, it seems to me that you need to look to team performance. I'd ask something like "how good was Team X at getting tossups, and how important was Player Y to Team X's tossup-getting ability?"

That idea from the player poll thread got me thinking: how exactly would you go about asking how good Team X was at getting tossups, and how important Player Y was to Team X's tossup-getting ability?

My first guess was that (player tossup PPG / team tossup PPG) * (team overall PPG / top team overall PPG) would be an interesting stat. Essentially, that stat asks "how good was this team relative to the rest of the teams, and what percentage of the tossups did the player get to contribute to that?" (I used team PPG instead of team tossup PPG in the team multiplier because it captures the fact that better teams are better at bonuses, and each tossup a player contributes comes along with the whole team's bonus prowess.) But it proved to be too similar to raw player PPG, and didn't yield many notable results.

My next guess was that squaring both team overall PPG and top team overall PPG would possibly yield more interesting results, since the top teams would be further separated from the weaker teams. (I got the idea of squaring the numerator and denominator from baseball's Pythagorean theorem win percentage formula.) The results of this were indeed interesting, because they jibed very closely with how I might subjectively rank the players.

I ran the numbers on the stats from the ACF Nationals top bracket and uploaded them to Google Sheets: https://docs.google.com/spreadsheets/d/ ... sp=sharing

(Note that several talented players, including Evan, Richard, Jacob, Trevor, Jordan, etc., are left off here because it's only the top bracket. I used only the top bracket because you'd have to use adjustments to rank players who didn't play common or comparable schedules.)

I'm not sure this tells you anything other than roughly how I think players could be ranked based on the relative strengths of their tossups performances and their teams' points, but I think it's still kind of cool.

[AKKOLADE]

Where would you rank with this?

(Okay, that joke is out of the way)

[Lightinfa]

Here's the formula applied to top bracket ICT, although with stats from all games at the moment instead of just top bracket games since it would take some time to separate that out.

https://docs.google.com/spreadsheets/d/ ... sp=sharing

[Sima Guang Hater]

It's conclusive guys. I'm #2!

[theMoMA]

Here is an attempt at a version that uses a correction factor (adjF on the sheet) between brackets: https://docs.google.com/spreadsheets/d/ ... sp=sharing

The sheet above ranks every player from Nats based on playoff stats alone.

Thanks to Adam Sperber for the general idea.

I calculated the correction factor by finding the average chance that a team in the particular bracket converts a tossup that the other team did not convert (either by getting it first or picking it up after a neg). (See this thread for the general framework; it's a PATH-like measure that I think is very good for comparing the relative strengths of two pools of teams.) Then I compared it to the top-bracket measure using the same square-over-square discounting as with the main stat. I used the square-over-square again because it seemed to give more empirically valid results. (I determined what I thought was more "empirically valid" by looking to similar teams in prelim stats that split into different playoff brackets.)

[theMoMA]

As a thought experiment, I wondered how putting four of the best players on a hypothetical superteam would affect their ratings. I figured a superteam of Matt Bollinger, Eric Mukherjee, Will Nediger, and Jordan Brownstein might be interesting to investigate (because all four are in the top five and would have their individual PPGs affected the most, and because they all fit the roles of a superteam that did quite well at CO 2012 composed of Matt, Eric, John Lawrence, and Chris Ray).

Using the 2012 CO and 2015 Nats stats as a guide, I estimated that the scoring breakdown might fall something like Matt 45, Eric, 35, Will 32, Jordan 30. (The 2012 CO stats are propped up by the fact that there were powers there, but were also driven down by the fact that CO is harder than Nats and it's harder to get questions against superteams than Nats teams, especially Nats teams where Will, Jordan, Eric, and Matt aren't all playing separately.)

I figured that this superteam playing 2015 Nationals could get about 15 tossups per game and average a shade over 20 points per bonus, for about 450 points per game (once you factor in 1-2 negs per game). I put those numbers into the spreadsheet, and here's what I got: Matt .519, Eric .454, Will .416, Jordan .390. Those numbers are basically exactly the same as they were when the four played separately, and would not change the order at all.

(I didn't change the "besttPPG," but that's just a scaling factor; changing it to 450 instead of 331.4 would yield the same relative rankings, just with smaller-looking numbers.)

I made a lot of assumptions to get these numbers (I picked them before running the numbers, by the way; I had no idea what would work to make the outcome similar to the initial numbers, other than I knew that ordering the scoring on the team the same as the order in the initial numbers would help), but I don't think that kind of scoring distribution on that potential superteam would be unreasonable. Of course, other reasonable estimates can lead to different numbers (if Will or Jordan can only score in the 20s, their ranking would fall to the back end of the top ten) but it illustrates that this stat might just be durable to team effects.

[Nabonidus]

All the same, using the Canadian players I actually know... your math ranks my teammate Joe Su over Jordan Palmer and Jay Misuk, both of whom are far better players than he is. A little further down, my other teammate Sebastian Drake (who only buzzes on inorganic chemistry and British military history), is ranked higher than Leslie Newcombe and Aayush Rajasekaran. That arrangement is just objectively wrong.

[Windows ME]

The list [the ACF one with all brackets] passes the eye-test really well. However, it seems like it is far too harsh to players playing in brackets outside of the Top 3. As above, I can certainly say that Jordan is a better player than Joe, yet his score is literally divided by two for not having made third bracket - I don't think such a large modification is necessary outside of the top two or three brackets because ACF Nationals questions are hard enough that if someone could theoretically beat up on the weakest teams by buzzing in at the end of most questions, they would not be in the bottom bracket to begin with due to the sheer number of tossups going dead!

Its a nice list though and a potential jumping off point for having a non-objective way to rank a large number of players (seems better to me than having people submit large ballots of fifty names which seems highly prone to exposure bias). Right now it seems to break down at lower ranks though.

[Fado Alexandrino]

Rankings are fun and everything, but I don't think ACF Nationals level questions are the best metric to rank players in like the 70s or above, considering how much variance there is after seeing how much of a cluster brackets three and beyond were.

There should be a consideration on how someone's performance jumps between brackets. How do you compare two people in the same low bracket, one who stayed at 20ppg and one who went from 10ppg to 30ppg after the rebracket?

[Peter13]

In response to your question Joe, I think the best answer is that no one knows. I would assume that the person who was more consistent between the rebracketing would be more of a specialist, while the person who jumped up in quality would likely be more of a generalist. I feel as the level of the game gets higher (like at ACF Nats), specialists are more sought after than generalists. But does that mean they are better players as a whole? I don't think anyone knows that answer.

This is what bothers me about using metrics to find the strength of players; it seems to either break down near the top when you try to make an "average player" baseline measurement, or breaks down near the bottom when you put only the top players in your set. Plus, it doesn't allow for variation with difficulty or changes in format, and sample size is usually too low to make anything but broad generalizations.

So I have a follow-up question: Do you think it is better to use metrics to pick the Top 25 Quizbowl Players, or use an aggregate of individuals' opinions to formulate it?

[theMoMA]

Yeah, I think it's roughly correct to say that the stat isn't super useful after the third bracket or so because the team shares are so small and can vary a lot depending on bracketing below that point. Perhaps there is a lower bound at which the correction factors don't matter all that much, probably because the teams are limited from scoring points more by their own knowledge rather than the knowledge of their opponents.

I will note, however, that the brackets were not handicapped based on their order; the only consideration that went into the adjustment factor was the strengths of the teams relative to the strengths of the top-bracket teams. In other words, the teams in the fourth bracket received the adjustment factor that they did entirely because of their performance as a group, not because their bracket was labeled "fourth" instead of "third." (As I said above, the stat I used was essentially: what is the percent chance that an average team in the bracket got a tossup that the other team did not convert? And as I said above, I find this a very intuitively sound measure for the relative strengths of groups of teams.)

Tying that together, it could well be that your opponents having a 45% chance rather than a 25% chance of picking up a given tossup that you fail to convert doesn't actually have much of an effect on how many points you score--at that point, you might be more limited by what you know than what your opponent does. I might try to do some regressions between preliminary and playoff stats of similarly skilled teams that end up in different brackets to see if there's a better way to calculate a bracket-correction factor.

Peter made some good points above, and to answer the general question that he poses, I think that having the players rank the players is more useful (at least now) because the only metrics that exist are applied to single tournament datasets. If there were a way to keep a season-long running tab on something like this "quick and dirty" measure, then perhaps that would give results similarly meaningful to what the player poll does now, but nothing like that really exists.

[minusfive]

There are multiple threads on this same subject, some of which involve perceptions of threats (don't go to a stand-your-ground state, Eric!), but I'm only going to say this once.
I am very flattered to be included in this colossal waste of time and effort. Having said that, this is a colossal waste of time and effort. We all (hopefully) play quizbowl to have fun and compete against other good players. To see myself being called "far better" than Joe is, even if true (it has not been proven), beside the point. There are things Joe knows really well that I don't (and presumably vice versa), and so even if "on paper" I am better/worse than another player, that doesn't predict any result; it also doesn't (shouldn't?) diminish my respect for my opponent. For instance, I enjoy playing against Derek and others, but I'd far rather play against them than get into an abstract dick-measuring contest. I'd rather think "hey, that was a good buzz of Derek's or man, Derek knows a lot about Milo of Croton, which is cool" than "I'm better than Derek or Derek is better than me." If I'm "worse" than Derek this encourages me to up my game, and we will both improve if there's strong competition. In sum: quizbowl playing fun; quizbowl pissing contest not fun.
Feel free to measure this rant against other rants to see how satisfied I should be with it.

[merv1618]

What does you or anyone else stand to gain from such a jabbing criticism though? Yes playing qb is fun, but as these forums clearly show so is the mania of forming hierarchies. In this case, Andrew threw some thought into an algorithm which churned out insanely qualitative results compared to his and others' past efforts. At the very least it's creating a neat little stats model, but in reality it's got some potential as an emotionless ranking system (given enough bonus variance if we're going between brackets).

Also I'm pretty sure a dick-measuring contest would yield incredibly incorrect results, especially with regards to women playing the game.