Stuff about bonus in high school sets

Dormant threads from the high school sections are preserved here.
Locked
User avatar
aestheteboy
Tidus
Posts: 570
Joined: Sat May 13, 2006 5:07 pm

Stuff about bonus in high school sets

Post by aestheteboy »

I'm going to describe a phenomenon which I notice in most high school sets. The problem, at least for me, is much less practical than aesthetic. In other words, it's a trivial issue, and most of what I say should be taken with light-heartedness.

Fluctuation in bonus difficulty is one of the most difficult problem to deal with. Because question sets are man-made, they are bound to have some imperfections. What irritates me, however, is that some writers seem not to understand the purpose of having bonuses - to distinguish the knowledge level of teams. To be more direct, some easy parts of the bonus that I heard this year are simply too easy. No one should ever ask in any part of the bonus (to use some exaggerated examples) who wrote King Lear and Macbeth, who was the first president on the United States, or what element has atomic number 1. They are horrible, for two simple reasons. First, they are pointless. Bonus parts with 100% conversion rate are meaningless, as are bonus with 0% conversion rate (at least, in high school sets). Second, it is unfair. Ridiculously easy bonuses are the quickest way to ensure difficulty fluctuation because it is impossible to make questions more answerable than the answers are accessible (unless you resort to cute clues).
The solution is clear. Don't write bonus parts that are too easy.
Daichi - Walter Johnson; Vanderbilt; U of Chicago.
Daichi's Law of High School Quizbowl: the frequency of posting in the Quizbowl Resource Center is proportional to the likelihood of being overrated.
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Stuff about bonus in high school sets

Post by Mechanical Beasts »

aestheteboy wrote:who wrote King Lear and Macbeth, who was the first president on the United States, or what element has atomic number 1.
I wouldn't be surprised if these are not converted by 100% of teams. The middle one is most egregious, but since people obviously don't all know it, surely four people taken together might not. The first one is hard if you haven't heard of those plays. Did I, as a freshman? Sure, I'd read them. I'd read them in sixth grade, too. But that's irrelevant: I bet on a nationwide level, a few teams will miss that bonus part. The last one is particularly hard if you don't know what atomic number is. Some people don't!

The bigger issue is that bonuses are far harder than tossups to make appropriate for a wide variety of skill levels. Tossups do it naturally; bonuses have four classes to sort teams into and if a set has to be used for everyone in the nation, then you're going to have some trouble. Given the perfect bonus that creates four equal categories, going by the Byko rankings, last year's Dorman and Whitman would have to be in the same class as Moline A. The next class would go down to Zionsville B, maybe, then another to J.E.B. Stuart A. You see what I mean.

Ultimately, what you're observing is this: it's very, very hard to write a set that is playable by 100% of the teams in the nation, and moreover, even supposing such a set existed--hey, it might--harder questions distinguish better teams better, and the problem is seen most dramatically in the part of the game that generally differentiates between teams the least well.
Andrew Watkins
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Stuff about bonus in high school sets

Post by Matt Weiner »

You would be surprised at what people do not know. Look at any tournament, high school or college, where this issue has come up, and you will see plenty of sub-10 bonus conversions.
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
User avatar
The Time Keeper
Auron
Posts: 1327
Joined: Sat Apr 19, 2003 9:26 pm
Location: Michigan

Re: Stuff about bonus in high school sets

Post by The Time Keeper »

Matt Weiner wrote:You would be surprised at what people do not know. Look at any tournament, high school or college, where this issue has come up, and you will see plenty of sub-10 bonus conversions.
This is definitely true, particularly at the HS level. It isn't as bad an issue if you're writing a set for one specific audience where you know exactly who will be playing, but for things like HSAPQ sets that are being used all over and will (hopefully) be increasingly introduced to areas not very familiar with good quizbowl, easy parts really can't be too easy. The key is making the easy parts of bonuses equally easy across the board so that medium or worse teams can't get screwed by a "hard" easy part. It is unfortunate that it leaves basically two bonus parts (and the tossups) to distinguish the good teams amongst one another, but the disparity in ability in hsqb right now is so huge that it's quite a task to make a set that can appeal to a very wide audience.
Pat Freeburn - No particular affiliation.
David Riley
Auron
Posts: 1401
Joined: Tue Apr 29, 2003 8:27 am
Location: Morton Grove, IL

Re: Stuff about bonus in high school sets

Post by David Riley »

Agree with all of the above. This really hit home for me when we tried packet submission at a past high school tournament--someone had written a "Who wrote King Lear.....?" etc. as the FINAL ROUND! It was edited out, but this team honestly thought they were writing a challenging bonus.
David Riley
Coach Emeritus, Loyola Academy, Wilmette, Illinois, 1993-2010
Steering Committee, IHSSBCA, 1996 -
Member, PACE, 2012 -

"This is 1183, of course we're barbarians" -- Eleanor of Aquitaine in "The Lion in Winter"
User avatar
Matthew D
Yuna
Posts: 920
Joined: Sun Oct 23, 2005 9:52 pm
Location: Scenic Grant Alabama

Re: Stuff about bonus in high school sets

Post by Matthew D »

well as someone that would like to learn to write better questions I wish we would have a writer's bootcamp that would be easier for some of us in the southeast to attend. Maybe someday
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Stuff about bonus in high school sets

Post by theMoMA »

I guess I will continue to disagree with this. It seems to me that we had this discussion last year, and I don't think your argument incorporates or even acknowledges the valid objections that were brought up last time you made it.

It really seems like your argument is uncharitable to non-elite teams and players. Believe it or not, there are quite a few teams who don't know what element is atomic number one. We do our best as writers to make one of the bonus parts something that as many teams as possible can get, which translates to 85-90%+ conversion.

Whether you get the easy part that asks "This guy wrote One Hundred Years of Solitude" (the 85% part, perhaps) or "This element with atomic number one" (the 99% part, perhaps) could certainly determine the game between two teams who are in that lower 10% category, but unfortunately, there is a threshold of knowledge implied by ideal conversion stats, and if you don't have the collective knowledge to pick up the ten-point part on most bonuses, you are indeed at the mercy of which bonuses you get.

I also disagree that the idea that bonuses are meaningless if one part has 100% conversion. It's not like these are free points available to the first team that can shout "Shakespeare" at random. You still have to get the tossup to receive the bonus.
Andrew Hart
Minnesota alum
User avatar
DumbJaques
Forums Staff: Administrator
Posts: 3109
Joined: Wed Apr 21, 2004 6:21 pm
Location: Columbus, OH

Re: Stuff about bonus in high school sets

Post by DumbJaques »

I think I mentioned in the discussion a few months ago that I also in general disagree with Daichi's point. An easy bonus part that talks about the plots of Cymbeline and Titus Andronicus and then says "name this author who also wrote King Lear" is both informative to those new teams (and maybe even some good teams) and convertible for most of the field, as an easy bonus part is meant to be. Perhaps Daichi has a different standard for easy bonus parts - I'm curious dude, do you mean that you think that greater than 85% of the field would still convert 10 points if the easy part asked for King Lear, or do you just think the minimum conversion threshold should be lower than 85% for the easy part? I'd argue, as Andrew does, that the former conclusion just isn't born out by looking at stats and considering all the teams in a field. If it's the latter, well, I guess that's something that can be more philosophically debated. But still, I don't think we incur a negative by making easy parts really easy (I mean, it's not as we're insulting people by saying "touch your nose" - at one point the fact that hydrogen was atomic number 1 was news to me, and that point was probably about the time I got into quizbowl).

I do think you can derive a valid argument from what I think is a flawed premise, and it's one I've been thinking about in regards to HSAPQ. If the easy parts are easy, which I and I think most people agree they should be (that is, a target of 80%+ conversion), we just have to accept that they'll get a nearly 100% conversion for the top teams every single time. If the middle parts are meant do differentiate between the middle teams, and hit a target of about 50% conversion (or even a little worse), then it seems to me that by definition the top teams are also going to be picking up basically 100% of these. In theory, that leaves you with only one bonus part to differentiate top teams. How hard do you make it? In a sense this is a real bind, because if you make it hard enough that not all top teams will convert, the likelihood of ANY middle teams getting it goes way down. Unlike college, were even a lot of middle teams can pick up 20 on most bonuses even when they're reasonably difficult, the difference between top and middle of the pack high school teams is kind of characterized at this point by markedly different canon awareness. If HSAPQ third parts ask things that really push those teams, as I kind of think someone should be doing, I'll posit that it's just entirely out of bounds for middle teams. Essentially what you're left with is the choice to pick middle parts that aren't too hard and rely on very difficult third parts to differentiate top teams (thus essentially giving a lot of middle teams 20 point bonuses) and inherently asking stuff that's difficult to balance as third parts, OR asking slightly more difficult middle parts (and perhaps screwing up middle games that end up being decided by who got the more blatantly easy easy part bonuses) and hard but not really hard third parts, which doesn't much push the top teams and would probably result in bonus conversion becoming less meaningful in those games.

I actually think HSAPQ handled this problem quite well in the one set I saw, which ended up having something of a middle ground between these two paths. Of course, that at times created some bonus imbalance that I noticed people mention as one of the (very few) areas of improvement for the first set.
Chris Ray
OSU
University of Chicago, 2016
University of Maryland, 2014
ACF, PACE
The Atom Strikes!
Tidus
Posts: 612
Joined: Wed Jun 13, 2007 7:05 pm
Location: Houston, Texas

Re: Stuff about bonus in high school sets

Post by The Atom Strikes! »

Actually, the larger complaints that I have about HSAPQ bonuses are that occasionally, they seem to lack a hard part altogether.
Henry Gorman, Wilmington Charter '09, Rice '13, PhD History Vanderbilt '1X
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Stuff about bonus in high school sets

Post by Matt Weiner »

Even if top teams are converting 30 on every bonus (which data suggest they are not) you can still distinguish them if the tossups are written properly.
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
evilmonkey
Yuna
Posts: 964
Joined: Mon Sep 25, 2006 11:23 am
Location: Durham, NC

Re: Stuff about bonus in high school sets

Post by evilmonkey »

theMoMA wrote: I also disagree that the idea that bonuses are meaningless if one part has 100% conversion. It's not like these are free points available to the first team that can shout "Shakespeare" at random. You still have to get the tossup to receive the bonus.
The reward for getting a tossup is 10 points and the OPPORTUNITY to answer a bonus. If you're going to write a part that is 100% convertible, you may as well just give twenty for the tossup, and have just the medium and hard parts of the bonus - this would shave 3-5 minutes off of games, and accomplish the same result.

But that doesn't really matter, because:
Matt Weiner wrote: You would be surprised at what people do not know.
Matt is exactly right here. Especially as you learn more, it shocks you when other people don't get the things that you consider second nature. I encourage you to go to a high school tournament, and watch a match between the two worst teams present. Such a display may shock you, and you'll want to scream at the kids for being so ignorant about things you think are commonplace, but it will help you get a better grasp on this.
Bryce Durgin
Culver Academies '07
University of Notre Dame '11
Texas A&M '15
leapfrog314
Wakka
Posts: 248
Joined: Mon Sep 12, 2005 10:49 am
Contact:

Re: Stuff about bonus in high school sets

Post by leapfrog314 »

evilmonkey wrote:I encourage you to go to a high school tournament, and watch a match between the two worst teams present.
Part of the reason I like quizbowl is that it encourages people to know stuff, even though the people in quizbowl would probably be making a point of knowing stuff regardless. But I have watched many a terrible match, and it is incredible to me how little some high school (and even college) students know about the most basic elements of Western culture (much less other cultures).
Carlo Angiuli, Indiana University
Director, Aegis Questions, Inc.
User avatar
Terrible Shorts Depot
Yuna
Posts: 760
Joined: Mon Mar 24, 2008 5:05 pm

Re: Stuff about bonus in high school sets

Post by Terrible Shorts Depot »

Part of me wants to say that easy parts should be harder, because some of them are insanely easy. Some are so simple that I would be shocked if anyone gets it wrong. However, some people do get them wrong, which leads me to the other part. We don't want to discourage teams from playing just because they are bad. Thus, I think that at least some easy bonus parts should trend closer to "Name this 1st president of the United States", as opposed to "Name this man who issued the Proclaimation of Neutrality".
Charlie Rosenthal
Shady Side Academy '09
Carleton College '13
University of Pennsylvania '18
User avatar
Matthew D
Yuna
Posts: 920
Joined: Sun Oct 23, 2005 9:52 pm
Location: Scenic Grant Alabama

Re: Stuff about bonus in high school sets

Post by Matthew D »

Please ya'll as a coach don't attempt to make it easier.. that is a very slippery slope till you end up with this mess of crap that might look something like quizbowl but is actually just Jeopardy for the retarded
Matt Dennis
Coach DAR Quizbowl Team
User avatar
cvdwightw
Auron
Posts: 3291
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Stuff about bonus in high school sets

Post by cvdwightw »

WARNING: TERRIBLE SPORTS ANALOGY AHEAD

The "easy" part of the bonus is like the extra point after a touchdown, only if the extra point were worth six points as well. It should be converted near 100% of the time by all competent people, but it's not converted at 100% by the best teams, and it's sometimes not converted at all by high school teams with terrible kickers. Saying that a team should just get 20 points for the tossup is like saying that a football team should get 7 points for a touchdown and not kick the extra point. Sure, that extra point requires minimal effort, but teams still find ways to screw it up. I've run into several bonuses thinking "any competent team will earn 30 points on this bonus" and watched an otherwise decent team earn nothing.

WARNING: MATH AHEAD

With hard parts, you have to remember that nationally elite teams are expected to convert 60-70% of the hard parts, with the idea being that the better team will convert slightly more - there is no floor effect (teams can't convert a part, so it doesn't matter) or ceiling effect (teams won't miss a part, so it doesn't matter) with the top teams and hard parts. Good teams are fully expected to get 20 points on about 95% of the bonuses and a full 30 on 40% of the bonuses; bump that latter number up to about 60% for the top teams. What this means is that we expect (0*0+10*0.03+20*0.34+30*0.63=) 26 ppb from nationally elite teams, 23 ppb from good teams, etc. A middle team who can convert 10% hard parts, 50% middle, 90% easy should get 15 ppb.

If we make the easy part harder, then we risk upsetting the bottom teams, where they are on average converting 70% of easy parts and 30% of middle parts for 10 ppb. If we make the hard part harder, then we bring down the top and middle teams' ppb, but the middle teams rarely get a 30 and the bottom teams are even less likely. If we make the middle part 10% harder for middle teams (bringing them down to 14.48), then we still fudge with the bottom teams (now below 30%) but can make the easy part easier to bring that back up (100-40-10 for a middle team is a 15 ppb and likely corresponds to a 80-20-0 = 10 ppb for a bottom team and a 100-80-40 = 22 ppb from good teams).

So we can see a 100-100-90-80-70 split for easy parts (87% conversion), 90-70-50-40-30 split for middle parts (56% conversion), and 40-20-10-5-0 split for hard parts (15% conversion), or we can see a 100-100-100-90-80 split for easy parts (94% conversion), a 80-60-40-30-20 split for middle parts (46% conversion), and a 40-20-10-5-0 split for hard parts (15% conversion). The former gives us a 23-19-15-12.5-10 while the latter gives us a 22-18-15-12-10 split from top to bottom quartile.

My point is that the easier you make the easy part, the harder you have to make the middle part to hit your benchmarks of 15 ppb for the median team and 10 ppb for the 90th percentile team. The easier you make your middle part, the harder you have to make your easy and hard parts to hit those benchmarks.

The reason 100% easy part conversion is bad is that you then need 0% middle part conversion by the 90% team to hit the 90th percentile team = 10 ppb benchmark. But, as you can see, we can hit 94% easy part conversion (only 6% of teams will find a way to screw it up) with decent-difficulty middle and hard parts and still hit the targets.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry
User avatar
Sir Thopas
Auron
Posts: 1330
Joined: Tue Mar 28, 2006 10:10 pm
Location: Hunter, NYC

Re: Stuff about bonus in high school sets

Post by Sir Thopas »

Good luck distinguishing between a 90% bonus part and a 94% bonus part, though.
Guy Tabachnick
Hunter '09
Brown '13

http://memoryofthisimpertinence.blogspot.com/
User avatar
cvdwightw
Auron
Posts: 3291
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Stuff about bonus in high school sets

Post by cvdwightw »

Sir Thopas wrote:Good luck distinguishing between a 90% bonus part and a 94% bonus part, though.
Frankly, I'd be surprised if 6% of teams managed to miss "Name this first president of the United States", but it's entirely possible, and it's better to err on the easy side because people will always convert less points than they deserve (the amount of points lost by failing to recall things one knows is typically greater than the amount of points gained by guessing).

And, if the same set is run many times across the nation, I'm pretty sure you can distinguish between a 90% and 94% bonus part (after the fact, of course). I'm sure if everyone sent exact scoresheets to NAQT detailing which tossups and bonus parts were converted and which were not, then NAQT could determine (and would spend time empirically determining) what is a 94% bonus part and what is a 90% bonus part.

Remember there are two distinct, but related questions about difficulty: what percentage of (easy/medium/hard) parts is a team likely to get given a certain ability level, and what percentage of teams are likely to get a certain (easy/medium/hard) part. If the 90th percentile team is scoring 50% on easy parts, but everyone above it is scoring 100%, then even though our overall conversion rate of easy parts (criterion II) is 90% I'd say that the questions are too difficult (because it fails criteria I, in that the 90th percentile team should be converting around 80-90% of easy parts). The easy part is written with the implicit assumption that teams above the 50th percentile will convert over 90% of easy parts regardless of actual difficulty, and we focus on getting the 90th percentile teams to convert 70-80% to base our idea of "easy part" difficulty. And yes, some of those teams are that dumb.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry
User avatar
Sir Thopas
Auron
Posts: 1330
Joined: Tue Mar 28, 2006 10:10 pm
Location: Hunter, NYC

Re: Stuff about bonus in high school sets

Post by Sir Thopas »

You sidestepped the exact point I was trying to make, which is that, yes, having percentages down to a tee are all well and good theoretically, but it's extremely difficult to do this as they're being written.
Guy Tabachnick
Hunter '09
Brown '13

http://memoryofthisimpertinence.blogspot.com/
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: Stuff about bonus in high school sets

Post by Sen. Estes Kefauver (D-TN) »

Yeah, Guy pretty clearly wasn't talking about after the fact statistical analysis but trying to figure out those statistic while one is writing the question.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
User avatar
cvdwightw
Auron
Posts: 3291
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Stuff about bonus in high school sets

Post by cvdwightw »

Jeremy Gibbs Free Energy wrote:Yeah, Guy pretty clearly wasn't talking about after the fact statistical analysis but trying to figure out those statistic while one is writing the question.
I disagree that these are unrelated. Quizbowl is by definition a feedback cycle. If a question is too hard it will be moved up the difficulty or thrown out. If a question is too easy it will be moved down the difficulty ladder. While we cannot say exact when we begin to write a question what its conversion will be, we can use empirical data we get to define whether a certain answer is "too hard", "too easy", or "just right" for what it's supposed to be.

If I write a bonus on William Shakespeare and find that 100% of teams are converting Shakespeare, 80% are converting Romeo and Juliet, and 60% are converting King Lear, then it's clearly not a good bonus regardless of how difficult I or anyone else I think it is. On the other hand, if 25% of teams are converting Shakespeare but 75% are converting Romeo and Juliet (maybe because I've used more obscure plays), then I know that I've screwed up in making Shakespeare the purported "easy" part and can change that the next time I or someone else writes/edits a Shakespeare bonus.

NAQT is always interested in conversion statistics, I know the EFT people sent out special EFT scoresheets last year to pinpoint exactly what was and was not answered (don't know if those statistics were ever made public), and I'd imagine HSAPQ would be interested in question conversion data as well. These are not for some masturbatory "look how hard I made my questions" reflection but for feedback on how hard or easy to make the next incarnation of that tournament. If NAQT can be shown that 80% of tossups on nonsensical answers go dead in 80% of rooms, then I'd virtually guarantee NAQT will stop letting those questions through because NAQT does not like dead tossups. If there's empirical data showing that 50% of teams are converting the "hard part" of an HSAPQ bonus, then it's likely that someone will contact the HSAPQ writer in question and tell him to make his hard parts harder. Using subjective categories like "too easy" and "too hard" just doesn't cut it at this level, especially when the top 1% of teams are the ones complaining that things are "too easy". That's not to say that the criticisms aren't valid, but that it's hard to write high school questions for players that are playing at an ACF Regionals level and not have them be "too easy" while making sure the South Podunk D team that doesn't know Washington from Lincoln can still get some points.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry
User avatar
Terrible Shorts Depot
Yuna
Posts: 760
Joined: Mon Mar 24, 2008 5:05 pm

Re: Stuff about bonus in high school sets

Post by Terrible Shorts Depot »

cvdwightw wrote:If I write a bonus on William Shakespeare and find that 100% of teams are converting Shakespeare, 80% are converting Romeo and Juliet, and 60% are converting King Lear, then it's clearly not a good bonus regardless of how difficult I or anyone else I think it is. On the other hand, if 25% of teams are converting Shakespeare but 75% are converting Romeo and Juliet (maybe because I've used more obscure plays), then I know that I've screwed up in making Shakespeare the purported "easy" part and can change that the next time I or someone else writes/edits a Shakespeare bonus.
Does your intention matter in writing the bonus? That is, if you intend for Part A to be easy (Romeo and Juliet in Dwight's example), but it ends up being hard and Part C (which you intend to be hard) ends up being easy, does it matter which part is which, as you as you have one easy, one medium, and one hard part?
Charlie Rosenthal
Shady Side Academy '09
Carleton College '13
University of Pennsylvania '18
User avatar
DumbJaques
Forums Staff: Administrator
Posts: 3109
Joined: Wed Apr 21, 2004 6:21 pm
Location: Columbus, OH

Re: Stuff about bonus in high school sets

Post by DumbJaques »

26 ppb from nationally elite teams, 23 ppb from good teams, etc. A middle team who can convert 10% hard parts, 50% middle, 90% easy should get 15 ppb.
Is this the standard we're accepting? If so, I'd say only a few high school events hit something near that each year (the first HSAPQ set, which I presume Dwight is using as a partial model for these numbers, hits them fairly perfectly). Of course, I wonder what you'd call a "nationally elite" vs. a "good" team in that field (or any field). If the definition is just teams who put up those corresponding numbers, it's obviously a redundant definition.

From what I remember of reverse-engineering HSNCT results, teams usually struggle to break 20 ppb, and I'd say that no college tournament I've ever been to hit anything close to that. So is there a lack of consensus about those numbers (and should they differ depending on the tournament being national/regular season)? Is every collegiate tournament a woeful disaster in bonus difficulty, or should the standard for college be different?
Chris Ray
OSU
University of Chicago, 2016
University of Maryland, 2014
ACF, PACE
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: Stuff about bonus in high school sets

Post by Sen. Estes Kefauver (D-TN) »

Dwight, you're totally missing the point! What I know I am saying, and am confident Guy is saying as well, is that when someone writes an easy part of a bonus, we can only make a ballpark estimate that 90% of teams will answer it, or whatever we're going for. Beyond very rough estimation, it is impossible to write a bonus that someone can confidently proclaim before it is played that "94% of teams will get this." All we can do is go "OK, I'm pretty sure this is easy enough that almost everyone should get this right," and then look at the statistics later to see if we met our goal, and if not, then we change our mind about what is convertable and try it differently the next time around. It is a fools errand to do anything beyond that while you are writing questions.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
User avatar
cvdwightw
Auron
Posts: 3291
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Stuff about bonus in high school sets

Post by cvdwightw »

Okay, I understand now that the point is that we cannot make implicit assumptions that within X%, Y% of teams will convert this part; I think you agree with the rest of my post that once we see how close we got to our target, we can tweak what is easy and what is hard.

Chris--I was working off the two assumptions that (1) most nationwide high school sets, whether NAQT, HSAPQ, or an independent set mirrored around the country, should see a median around 15 ppb; and (2) if a middling team is converting half the middle parts, then a team that is competitive on ACF Regionals level packets should be converting around 90% of middle parts.

At a regular season level, the variance between teams' skill levels is huge - on one hand we have the national competitors and on the other we have teams that haven't heard of Shakespeare. At the nationals level, this variance is much less - the median team has a much greater skill level and the top and bottom teams are not as distant, so the best teams hit a 100-70-30 for 20 ppb with the 90-50-10 model instead of 100-90-70 for 26 ppb, and the bottom teams hit a 80-20-0 for 10 ppb instead of the 70-30-0.

For fields where the top of the field is significantly better than the median team, we should not be surprised with top teams at ~25 ppb. At ACF Fall, I would expect nationally elite teams to score around 25 ppb; the reason we don't see this is because most of those teams don't play ACF Fall, or at least don't play their full nationals lineup. At ACF Nats, the variance in team ability is much smaller, and the top teams' BC should accordingly be brought down.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry
User avatar
Judson Laipply
Rikku
Posts: 492
Joined: Sat May 05, 2007 10:02 pm
Location: Bucyrus, Ohio

Re: Stuff about bonus in high school sets

Post by Judson Laipply »

I think that a bigger problem is bonuses that have a 0-10% conversion across the board for all teams from Charter A to Charter Z. What is the point of having the metaphorical extra point if you are kicking it from midfield, your own 20 and then your own endzone?
James L.
Kellenberg '10
UPenn '14
UChicago '20
User avatar
Sir Thopas
Auron
Posts: 1330
Joined: Tue Mar 28, 2006 10:10 pm
Location: Hunter, NYC

Re: Stuff about bonus in high school sets

Post by Sir Thopas »

Glacierguy1 wrote:I think that a bigger problem is bonuses that have a 0-10% conversion across the board for all teams from Charter A to Charter Z. What is the point of having the metaphorical extra point if you are kicking it from midfield, your own 20 and then your own endzone?
There are none of those this side of ACF Regionals. Trust me on this one.

Unless you're counting that one awful, awful bonus in IS-80A.
Guy Tabachnick
Hunter '09
Brown '13

http://memoryofthisimpertinence.blogspot.com/
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Stuff about bonus in high school sets

Post by Mechanical Beasts »

Sir Thopas wrote:
Glacierguy1 wrote:I think that a bigger problem is bonuses that have a 0-10% conversion across the board for all teams from Charter A to Charter Z. What is the point of having the metaphorical extra point if you are kicking it from midfield, your own 20 and then your own endzone?
There are none of those this side of ACF Regionals. Trust me on this one.

Unless you're counting that one awful, awful bonus in IS-80A.
Yeah, the easiest class of such bonuses would get thirtied 1/1000 of the time by the best teams in the nation, according to your argument. I played with Henry over the summer, and with Dan Puma, and against Naren and Guy, for a decent sample. I'd have a hard time imagining a bonus you can say confidently that they would have a terrible chance at tenning.
Andrew Watkins
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: Stuff about bonus in high school sets

Post by Sen. Estes Kefauver (D-TN) »

I can, if its a bad bonus.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Stuff about bonus in high school sets

Post by Mechanical Beasts »

You'd have to find a thousand of them and they'd have to thirty one of them. I'm saying he's overstating the probability. Will they probably zero Baylis-Hillman reaction/ Darzens reaction/ Reformatsky reaction? Yeah, and I would have tenned it before this week. But I'm betting that if a bonus is simply stupid or ill-conceived--as opposed to intentionally targeted at a high difficulty--it's totally not going to be zeroed that frequently.
Andrew Watkins
User avatar
Judson Laipply
Rikku
Posts: 492
Joined: Sat May 05, 2007 10:02 pm
Location: Bucyrus, Ohio

Re: Stuff about bonus in high school sets

Post by Judson Laipply »

Sir Thopas wrote:
Glacierguy1 wrote:I think that a bigger problem is bonuses that have a 0-10% conversion across the board for all teams from Charter A to Charter Z. What is the point of having the metaphorical extra point if you are kicking it from midfield, your own 20 and then your own endzone?
There are none of those this side of ACF Regionals. Trust me on this one.

Unless you're counting that one awful, awful bonus in IS-80A.
I am specifically counting that one in IS 80A.

And there are bonuses that are at such a high level that teams below the top of the highest tier of quizbowl would just barely have a hope of 10ing.
James L.
Kellenberg '10
UPenn '14
UChicago '20
User avatar
Sir Thopas
Auron
Posts: 1330
Joined: Tue Mar 28, 2006 10:10 pm
Location: Hunter, NYC

Re: Stuff about bonus in high school sets

Post by Sir Thopas »

Glacierguy1 wrote:I am specifically counting that one in IS 80A.

And there are bonuses that are at such a high level that teams below the top of the highest tier of quizbowl would just barely have a hope of 10ing.
Not in high school packets, there aren't.
Guy Tabachnick
Hunter '09
Brown '13

http://memoryofthisimpertinence.blogspot.com/
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: Stuff about bonus in high school sets

Post by Sen. Estes Kefauver (D-TN) »

Yeah, I would have to agree that there are probably some bonuses at ACF regionals or above level that the best high school teams could only 10, but I really don't think any well written, appropriate bonus for amy level below that will be that hard.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
Locked