Side-by-side stats for 2010-11 sets

[Grams's Go-Go Boots]

I put together an Excel sheet that puts the average, median, 3rd quartile, 1st quartile, standard deviation, max, and min PPB values of the various high school quiz bowl sets from last year side-by-side. This includes every stat line that I could find for these tournaments.

The sets included are:

BATE
BHSAT
DAFT
DCCAT
Delta Burke
FKT
GSAC
HFT
HT15
HT16
HT17
IMSANITY
IS96
IS97A
IS98
IS99A
IS100
IS101A
IS102
IS103A
IS104
IS105
LIST
MD Spring
Prison Bowl
PSATTPOT
RTO
SCOP
VCUVandy
WUHSAC

The spreadsheet should default to the overall view.

I excluded stats that did not include PPB for obvious reason. I also excluded cases like "team gets 5 ppb, only has one bonus, 30s it" to avoid skewing the data; there was only one or two cases of this.

I might have/probably missed some data, particularly tournaments without teams that would make my "best teams watch list" for hsqbrank and JV tournaments. I did not include middle school only events/divisions, because that doesn't quite seem to be the point.

[Down and out in Quintana Roo]

This is pretty ridiculously awesome stuff.

One question, and i guess this is best perhaps addressed to Jeff, is regarding Fred's "5 point spread pie" chart in the last tab of the sheet. When it comes to NAQT IS sets, is it NAQT's goal to achieve as much of a perfect "bell curve" on PPB as possible?

For example, Fred's data shows that

PPB Range #
0.00-4.99 51
5.00-9.99 381
10.00-14.99 479
15.00-19.99 334
20.00-24.99 114
25.00-30.00 2

Obviously NAQT would want the higher two categories to rise in number, but is there a specific goal that they're looking for in their sets? Certainly, to me, there's a problem with only 2 teams, in the entire country, being able to reach over 25ppb on all the IS sets combined, yet more than 50 teams could not even reach 5ppb. Or, to take that even further, it's still a problem to me that less than 115 teams, in the entire country, could break 20ppb, but that almost 400 teams could not reach 10ppb.

[bt_green_warbler]

This is awesome work, Fred.

As for theory-of-bonus-conversion:

0. Obviously it's a good idea to make our bonuses easier; of course we would like to see the number of teams in those highest categories increase and improve the overall averages by a couple of points.

That being said, I don't think it's possible to generate a true bell curve with even tails on both sides. Some explanations:

1. Teams on the far left end will always be more numerous than teams on the far right, because "let's find four interested freshmen who've never played before so we can field an E team" is easier than "let's build a national championship-quality team that can put up 26 ppb."

2. The ceiling on high bonus conversions is more limiting than the floor on low bonus conversions. Even awesome teams will miss bonus parts sometimes because they mishear the moderator, disagree while conferring, or just plain get a bonus in some subcategory the team doesn't know well enough to earn all 30 points. Furthermore, a team putting up very high bonus conversions can always leave and go play something more challenging (up to and including "let's go to CO where no team in history has ever broken 25 ppb!"). We don't know what full-strength State College A would have converted on an IS set in 2011; probably something pretty ridiculous, given the 66/34/36 and 24.5 ppb stat line they assembled with only three players. By contrast, it's easy to find brand-new teams that put up 3 to 4 ppb on introductory sets. Those teams don't have anywhere else to go because the alternative to playing novice sets is not playing. (And beyond a certain bound, editors run into problems making the easy parts of bonuses that much easier, as anyone who has ever written the last few bonuses of a middle school set can attest.)

I would be interested, by the way, to see what the highest all-time bonus conversion figure is for any team at any level.

[Grams's Go-Go Boots]

It should be noted that I completely forgot that 5 point split pie chart tab thing was on the sheet before I uploaded it (I think it's a remnant from the "all IS stats" sheet). So it has no new info.

[ether a go-go]

Fred is even more awesome than I thought.

I wonder how difficult it would be to perform Excelsior rankings on this data to correct for field quality. Basically, you would take all the bonus conversions for a single team during the year and pair them up into matches. If a team appears four times in the spreadsheet, then you would get a total of six matches between different combinations of the sets they played on. The result might be a good measure of which set was the easiest. It wouldn't be a perfect comparison because team composition varies from week to week, but given the amount of data that sort of noise might not be so bad.

One of the problems with just using the front page numbers is that some sets were OK at some sites and not at others. For example, the overall numbers for BATE are bad, but if you look at the raw data you see that it played well at two or three sites and poorly at two or three sites. So, the conclusion shouldn't be that BATE is bad, it should be that it's more appropriate for stronger fields, assuming that this year's set is similar to last year's. I think this data is useful to tournaments picking sets, though that job is still complicated by the fact that front page data is a little misleading because it is very dependent on last year's field strengths.

It seems like a goal would be to have a median around 15 and the quartiles somewhere around 11 and 19, which would lead to a standard deviation around 6. Several sets were around those numbers, though several were not. It's difficult to hit those numbers, in part because often a set is written without knowing what field will play on it, especially the fields at mirrors. It's also important to keep in mind that whether or not a set hits the right numbers is not the best measure of whether or not a set was good or not--it has to have interesting clues about interesting topics, pyramidal tossups with accessible answers, and bonuses of consistent difficulty, all of which are more important than appropriate bonus conversion but more difficult to measure.

One interesting comparison is that IS104 and IS105 had better numbers than the earlier IS sets. I'm wondering if that's just random noise, a difference in field strength, or a reflection of NAQT looking at some of the low earlier numbers and trying to make their IS Sets a little easier as the year went on.

[Excelsior (smack)]

I wonder how difficult it would be to perform Excelsior rankings on this data to correct for field quality. Basically, you would take all the bonus conversions for a single team during the year and pair them up into matches. If a team appears four times in the spreadsheet, then you would get a total of six matches between different combinations of the sets they played on. The result might be a good measure of which set was the easiest. It wouldn't be a perfect comparison because team composition varies from week to week, but given the amount of data that sort of noise might not be so bad.

Let me see if I'm understanding this. Let's say that school X played on the following sets with the following PPBs:
IS-103A: 22
HSAPQ-18: 19
IS-105
HFT: 12

Then, we would create matches between each of these sets using bonus conversion as "match scores", so that IS-103A has three wins, HSAPQ-18 has two wins, IS-105 has one win, and HFT has zero wins. We do this for all the data that Fred has, and then we run this through KRACH. This would give us a ranking in which the top-ranked set is the easiest one of the year, while the bottom-ranked one is the hardest. Does this sound right? If so, I can implement this (though this seems to me like it would be more of a historical curiosity than anything else).

[Excelsior (smack)]

Ranking of 2010-11 sets by difficulty

I did this. In addition to noise from changing lineups and such, there is also noise due to school-naming shenanigans (umpteen schools named University, etc). I went through and made a bit of an effort to rename teams when there were obvious changes to be made (DCD <-> Detroit Country Day, etc), but I was obviously not able to do this completely.

[Mewto55555]

It seems like there are disproportionately many sets from later in the year near the top (perhaps due to teams improving as the year went on?)

[ether a go-go]

Thanks, Ashvin. That's pretty cool.

While tournaments near the end of the year do have a natural advantage due to teams getting better, I think this also had something to do with people deciding during last year that sets were too hard in general and then purposely making sets like LIST and VCUBilt more accessible.

Did RTO use the same scoring system as everybody else? If those numbers are out of 30 points, then they are extremely low.

[Grams's Go-Go Boots]

One of the problems with just using the front page numbers is that some sets were OK at some sites and not at others. For example, the overall numbers for BATE are bad, but if you look at the raw data you see that it played well at two or three sites and poorly at two or three sites. So, the conclusion shouldn't be that BATE is bad, it should be that it's more appropriate for stronger fields, assuming that this year's set is similar to last year's. I think this data is useful to tournaments picking sets, though that job is still complicated by the fact that front page data is a little misleading because it is very dependent on last year's field strengths.

My impression of BATE from the data alone is that it was somewhat difficult, but not ridiculously so, for top teams (21st out of 30 sets in 1st quartile), but for teams below that tier, it was too hard (29th out of 30 in both median and 3rd quartile ppb).

However, I certainly wouldn't advise looking at the data in a vacuum.

[No Electricity Required]

Ranking of 2010-11 sets by difficulty

I did this. In addition to noise from changing lineups and such, there is also noise due to school-naming shenanigans (umpteen schools named University, etc). I went through and made a bit of an effort to rename teams when there were obvious changes to be made (DCD <-> Detroit Country Day, etc), but I was obviously not able to do this completely.

Is there a reason GSAC is not in this list?

[Excelsior (smack)]

Is there a reason GSAC is not in this list?

Oops. Fixed.

[Francis the Talking France]

Thanks, Ashvin. That's pretty cool.

While tournaments near the end of the year do have a natural advantage due to teams getting better, I think this also had something to do with people deciding during last year that sets were too hard in general and then purposely making sets like LIST and VCUBilt more accessible.

Did RTO use the same scoring system as everybody else? If those numbers are out of 30 points, then they are extremely low.

RTO was actually that hard I think..Even the top teams we had there were struggling mightily with the bonuses. The tossups were fine though, I think.

[rchschem]

Did RTO use the same scoring system as everybody else? If those numbers are out of 30 points, then they are extremely low.

RTO was actually that hard I think..Even the top teams we had there were struggling mightily with the bonuses. The tossups were fine though, I think.

I'm gonna go with weak field on that one. I would not feel good about comparing our questions on the basis of points put up at the tournament. No offense to the teams involved, it's just where we are right now.

Eric

[Francis the Talking France]

Every team aside from TJ Classical that attended the HSNCT had a lower PPB at RTO than at the HSNCT though. I think it's a combination of a weak field and a tough set.

[Broad-tailed Grassbird]

This is pretty ridiculously awesome stuff.

One question, and i guess this is best perhaps addressed to Jeff, is regarding Fred's "5 point spread pie" chart in the last tab of the sheet. When it comes to NAQT IS sets, is it NAQT's goal to achieve as much of a perfect "bell curve" on PPB as possible?

For example, Fred's data shows that

PPB Range #
0.00-4.99 51
5.00-9.99 381
10.00-14.99 479
15.00-19.99 334
20.00-24.99 114
25.00-30.00 2

Obviously NAQT would want the higher two categories to rise in number, but is there a specific goal that they're looking for in their sets? Certainly, to me, there's a problem with only 2 teams, in the entire country, being able to reach over 25ppb on all the IS sets combined, yet more than 50 teams could not even reach 5ppb. Or, to take that even further, it's still a problem to me that less than 115 teams, in the entire country, could break 20ppb, but that almost 400 teams could not reach 10ppb.

I get what you are saying Andrew, but it's really hard to make the 0-5 group get smaller without truly giving away points. I checked from the past MSU tournaments, and the teams that actually pulled this off were all getting less than 15 tossups in the whole day. That's truly miserable, but often times it's 2 freshmen who have never played before. Their knowledge base is smaller than the average quiz bowler to begin with, and with the diverse subjects quiz bowl has, these freshmen probably don't anything about the bonuses they actually get.

As for getting above 25 points in a tournament. That is a truly remarkable feat. That means you average less than one wrong bonus part for every two tossups you get. Over the course of a tournament a team with a ~25 PPB is going to get tons and tons and tons of bonuses. High School quizbowlers, even the top ones do suffer from fatigue. And eventually even the best of teams will zero a bonus on occasion. You can only get so many 10s in a tournament before it knocks you down to a 22-23 PPB. It would be nice to make a true bell curve, but for competition purposes, the current level of bonus difficulty at NAQT tournaments is ideal for the top 60-70% of competition. The bottom 30% are probably a little in over their heads, but the best way to get better is to play questions a little above your level, especially at that age.

[Tower Monarch]

I get what you are saying Andrew, but it's really hard to make the 0-5 group get smaller without truly giving away points. I checked from the past MSU tournaments, and the teams that actually pulled this off were all getting less than 15 tossups in the whole day. That's truly miserable, but often times it's 2 freshmen who have never played before. Their knowledge base is smaller than the average quiz bowler to begin with, and with the diverse subjects quiz bowl has, these freshmen probably don't anything about the bonuses they actually get.

As for getting above 25 points in a tournament. That is a truly remarkable feat. That means you average less than one wrong bonus part for every two tossups you get. Over the course of a tournament a team with a ~25 PPB is going to get tons and tons and tons of bonuses. High School quizbowlers, even the top ones do suffer from fatigue. And eventually even the best of teams will zero a bonus on occasion. You can only get so many 10s in a tournament before it knocks you down to a 22-23 PPB. It would be nice to make a true bell curve, but for competition purposes, the current level of bonus difficulty at NAQT tournaments is ideal for the top 60-70% of competition. The bottom 30% are probably a little in over their heads, but the best way to get better is to play questions a little above your level, especially at that age.

I agree with what you're saying overall, but I think it's a completely reasonable goal to ensure that teams with 20+PBB never 0 a bonus, so making the easy part a notch closer to "truly giving away points" would allow for a slight increase in the top and bottom group while shifting the plurality from 10-15 to 15-20, which to me is preferable.

[Broad-tailed Grassbird]

I get what you are saying Andrew, but it's really hard to make the 0-5 group get smaller without truly giving away points. I checked from the past MSU tournaments, and the teams that actually pulled this off were all getting less than 15 tossups in the whole day. That's truly miserable, but often times it's 2 freshmen who have never played before. Their knowledge base is smaller than the average quiz bowler to begin with, and with the diverse subjects quiz bowl has, these freshmen probably don't anything about the bonuses they actually get.

As for getting above 25 points in a tournament. That is a truly remarkable feat. That means you average less than one wrong bonus part for every two tossups you get. Over the course of a tournament a team with a ~25 PPB is going to get tons and tons and tons of bonuses. High School quizbowlers, even the top ones do suffer from fatigue. And eventually even the best of teams will zero a bonus on occasion. You can only get so many 10s in a tournament before it knocks you down to a 22-23 PPB. It would be nice to make a true bell curve, but for competition purposes, the current level of bonus difficulty at NAQT tournaments is ideal for the top 60-70% of competition. The bottom 30% are probably a little in over their heads, but the best way to get better is to play questions a little above your level, especially at that age.

I agree with what you're saying overall, but I think it's a completely reasonable goal to ensure that teams with 20+PBB never 0 a bonus, so making the easy part a notch closer to "truly giving away points" would allow for a slight increase in the top and bottom group while shifting the plurality from 10-15 to 15-20, which to me is preferable.

This is true, but we're moving away from this because the "stock bonus" is frowned upon. I think everyone would rather see a nationals level team zero a bonus every now and then instead of having more stock bonuses.

[Grams's Go-Go Boots]

I think everyone would rather see a nationals level team zero a bonus every now and then instead of having more stock bonuses.

I doubt anyone who is putting together a "regular difficulty" or easier set wants this, since if there are bonuses present that top teams are going to 0, then the hundreds of teams in lower tiers are going to be 0'ing lots more bonuses.

[ether a go-go]

I can see an elite team zeroing an occasional Pop Culture or Computational bonus if you use them, because it's possible for elite teams to be bad at those things and those questions can be quirky. However, if a legitimately very good team zeroes a Lit, Science, History, Fine Arts, or Social Science bonus, then it's pretty close to a certainty (though not a complete certainty) that the bonus is very bad.

[in on these shenanigans]

I feel like more people need to post to the effect of "Nalin's opinion is wrong and dangerous," so that up and coming question writers don't stumble here and get the idea that an impossible bonus is something "everyone would rather see."

In the interest of adding to the discussion-- What exactly do you mean by "stock bonus," in what way(s) is it being frowned upon, and by whom?

[Horned Screamer]

Nalin, where on earth do you come up with your awful opinions?

[Grams's Go-Go Boots]

I feel pretty confident saying this: any idea that we need to make anything at the high school level that aims to appeal to more than just the top 40 or so teams* more difficult is absolutely wrong, and there's data that supports that.

So, yes, Nalin's post is wrong. To say the least.

*so, basically, no tournaments, unless you want to include some of the regular season sets, because even nationals want to appeal to more than just those teams

[bt_green_warbler]

This is true, but we're moving away from this because the "stock bonus" is frowned upon. I think everyone would rather see a nationals level team zero a bonus every now and then instead of having more stock bonuses.

Let's assume for the moment that there is concern (somewhere) about bonuses becoming "stock" because they repeat the same three parts over and over again.

What would be the right way to fix this problem? The correct solution is "write about different subjects for the hard part of the bonus." "Deliberately writing a bonus that every team in the field will zero" is a terrible idea. (And both of the above situations are to be distinguished from "this is an otherwise very good team that earns zero on a particular bonus because they don't know economics/African literature/earth science.")

[ether a go-go]

This point may be pointless, but I'm going to make it anyways.

When we look at this data, we are analyzing it per team as opposed to analyzing it per question, and that lowers the statistics.

As a simple example as to how this works, let's say that State College played a match against New Trier B and that State College answered 18 TU and converted 25 PPB, while New Trier B answered 2 TU and converted 5 PPB. (I don't know how New Trier B would get those 2 TU, but bear with me.) The way we normally look at stats, and the way this spreadsheet does it, we would say that the average bonus conversion was 15 PPB. However, during the match there were 60 bonus parts asked and 46 answered correctly, which corresponds to 23 PPB.

For an actual example, I looked at the Greenville Comet last year, which used HSAPQ. If you average the PPBs for each team, which is the stat that shows up in this spreadsheet, you get 11.3 PPB. However, there were 15350 bonus points scored from 1211 bonuses asked during the tournament, which gives 12.7 PPB. In general, I've always thought that 15 PPB was a good goal for a high school tournament, but I don't know which of these two numbers is supposed to be 15 PPB. If it's the former, then the set was significantly too difficult for the field. If it's the latter, then the set was a little too difficult for the field.