"Regular" High School Difficulty

Dormant threads from the high school sections are preserved here.
User avatar
AlphaQuizBowler
Tidus
Posts: 695
Joined: Mon Dec 03, 2007 6:31 pm
Location: Alpharetta, GA

"Regular" High School Difficulty

Post by AlphaQuizBowler »

What is "regular" high school difficulty?

This question is the main thing I've been thinking about since playing the LIST set, and I feel like this important issue has been lost behind some of the more specific criticisms of the set that I've made.

I feel like it's unhelpful for everybody to try to produce a "regular" high school set. What I mean is this: there are a lot of tournaments that claim to be regular high school difficulty, and the term ends up encompassing so much of a difference in difficulties that it doesn't really mean much at all. Prison Bowl was "regular difficulty," as was BATE and LIST. NAQT IS-sets and HSAPQ sets are presumably regular difficulty, and PSAT aimed for "just about the difficulty of a vendor set." HFT was advertised as "barely harder than a regular-difficulty tournament." A look at the PPBs and power rates for these tournaments shows the discrepancies inherent in using such terminology.

The college quizbowl season has a variety of difficulties throughout the season: there's Collegiate Novice on the very low end; ACF Fall, MUT, and EFT (though all with their own characteristics) on the "easy" side; Penn Bowl, ACF Regs, and others at collegiate "regular" difficulty; then national and post-national levels. This variety makes it all the more difficult to understand why some of the main proponents of the "one-difficulty-fits-all" model for high school difficulty are experienced college players.

What I'm interested in exploring is a better-delineated concept of high school difficulty. There is a wide spectrum of team ability at the high school level, and to me that means there is room for at least a few different levels of high school difficulty. New teams or teams that don't practice often would obviously be put off if every high school tournament was NSC-level, but at the same time teams that have a lot of knowledge and put a lot of effort into quizbowl want to be challenged. There is a need for sets like LIST, to be sure, but there is also a need for better explaining to players what to expect when they play it. And from my experiences at a mirror of HFT 2008, I can tell you I certainly don't want to return to a difficulty level where the top teams at a tournament have final scores like 140-80 and PPBs below 15.

There's definitely space in the high school season for tournaments like Fall Novice, and it would be great to have another Fall-Novice-level tournament in the spring (along the lines of HAVOC from a few years ago). To me, LIST and recent HSAPQ sets are what I would classify as the "easy" of high school level. Which isn't a problem, as I think the writers make a concerted (and largely successful) effort to make those tournaments extremely accessible. Tournaments like GSAC and BATE represent my ideal of "regular difficulty" high school sets. And beyond that? That's another point for discussion. I would say that, in addition to top teams playing college sets, there is room in the high school season for 1 or 2 nationals warmup events on more challenging questions (along the lines of recent high school tournaments run on DII SCT, but mACF questions written for high schoolers, like the Weekend of Quizbowl Saturday event).

The above paragraph, though, is all suggestion, as I don't claim to have enough knowledge or experience to define high school difficulty. I just hope this thread can lead to a better dialogue about what different high school teams want (and need) out of high school questions.
William
Alpharetta High School '11
Harvard '15
User avatar
Steeve Ho You Fat
Auron
Posts: 1138
Joined: Mon Jun 01, 2009 11:48 pm

Re: "Regular" High School Difficulty

Post by Steeve Ho You Fat »

I wouldn't really say that the production of sets now is a problem, but perhaps the naming. For example, looking at high school sets produced this year, the only ones I can think of that were explicitly billed as "easier than regular difficulty" were NAQT A sets, Fall Novice, and Fall Kickoff. That means that sets from the difficulty of LIST and PSAT to DAFT, BATE, and GSAC, all advertised themselves at regular difficulty. Seemingly, this would be a problem. However, if you actually look at it, there are a really wide variety of sets out there. At the easier end, there are LIST and PSAT. Passing through HSAPQ Sets, IS Sets, and Prison Bowl, you start to get to BATE and GSAC, and eventually DAFT and HFT. Is there a problem with this distribution of difficulty? Not at all, I'd say: there's plenty of more accessible, but not novice stuff for newer players/programs, some stuff right in the middle, and some stuff that can challenge even very good players.

However, almost all of these sets said that they were "regular difficulty." LIST said that its tossups would have easier answerlines, but that was it, and HFT said they would be "barely" above regular difficulty. What tournament producers need to do is better describe their sets first. If William had known that LIST was going to be on the easy side, maybe he would have decided not to play it. Likewise, because I knew HFT would be on the difficult side, I decided to go to extra effort to get a chance to play it. Perhaps instead of novice-regular-nationals prep-nationals, we could think of (and advertise) our tournaments on a scale more along the lines of novice-easy regular-regular-more challenging regular-nationals prep-nationals.

In college, where difficulty is better defined and quantized, for lack of a better word, the better players know "Oh hey, if I go to ACF Fall I'll just be complaining about how it was too easy afterwords, I better not go," while you don't see a lot of novice teams showing up at ACF Nats. Making sure that everyone going to a tournament knows exactly what they're getting into would go a long way towards solving your problem.
Joe Nutter
PACE Emeritus
Michigan State University '14
Walnut Hills High School '11
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: "Regular" High School Difficulty

Post by Mechanical Beasts »

Joe N wrote:Perhaps instead of novice-regular-nationals prep-nationals, we could think of (and advertise) our tournaments on a scale more along the lines of novice-easy regular-regular-more challenging regular-nationals prep-nationals.

In college, where difficulty is better defined and quantized, for lack of a better word, the better players know "Oh hey, if I go to ACF Fall I'll just be complaining about how it was too easy afterwords, I better not go," while you don't see a lot of novice teams showing up at ACF Nats. Making sure that everyone going to a tournament knows exactly what they're getting into would go a long way towards solving your problem.
One of the reasons for the greater quantization is probably that college has a greater absolute and relative spread in difficulty. The canon stretches perhaps fiftyfold, rather than twofold.
Andrew Watkins
User avatar
Boeing X-20, Please!
Rikku
Posts: 406
Joined: Sun Jul 26, 2009 2:40 pm
Location: Evanston, IL

Re: "Regular" High School Difficulty

Post by Boeing X-20, Please! »

I'm going to say that I fully support what William Horton is saying here. I agree with the subsequent posters that there already is to somewhat a variety of difficulty levels and it would be beneficial to attempt to better categorize these, but I think that's not going far enough to truly mitigate the problem that William is describing.

In addition, I also think that it might be good if as a community, we tried to organize ourselves closer to the way that college quizbowl does it, with the tournaments on the year getting progressively harder since presumably teams get better as the year goes on: You have the ACF Falls and College Novices in the beginning of the year, you have some regular difficulty tournaments in the middle of the year, and then you get some opens and nationals in the Spring. Sure, there's some tournaments that go against this (e.g. MUT in the Spring) and that's more than fine, but I think it would be worth it make a concentrated effort to emulate this.
Nolan Winkler
Loyola Academy '12
UChicago '16
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: "Regular" High School Difficulty

Post by AKKOLADE »

All right, so, let's talk about tournaments here. I'm looking at the stats for IS-96. Using the stats from NAQT.com*, 13 tournaments used IS-96 and tracked ppb: Cal Classic, DAR Invitational, GINVIT, Maryland Fall, Missouri Fall, Olmsted Falls Invitational, Quaker Bowl, SAGACITY, Samhain, TQBA Kickoff, TWAIN XI, University High School, and Wisconsin Fall.

*Except for the Missouri Fall & University High School tournaments, which used bouncebacks and as such don't have ppbs listed on naqt.com; for those, I used the combined stats made available by the hosts.

Using this, 271 teams produced stat lines.

The average ppb is 11.95.
The median ppb is 11.92.
The third quartile ppb is 15.09
The first quartile ppb is 8.19.
If you adjust for all the teams that are not Bellarmine A, the average ppb is 25.04.

Percentage of PPBs in the following ranges:

25.00-30.00: 0.37% (1)
20.00-24.99: 6.64% (18)
15.00-19.99: 18.82% (51)
10.00-14.99: 36.53% (99)
5.00-9.99: 30.26% (82)
0.00-4.99: 7.38% (20)

A set intended to be normal difficulty for high school sees 74.17% of teams fail to average 15 ppb. I think it's safe to say that, if you're willing to accept the assumption that IS-96 is representative, in the general sense, of tournaments intended to be normal difficulty for high school teams and the success of accomplishing that, these tournaments are too difficult (unless having the average ppb be well below 15 is acceptable, but I heavily disagree with that).

Let's also consider the reality of producing these sets by looking at the two companies that produce the most regular difficulty tournaments: NAQT and HSAPQ. Note that this is my recollection of the situation with these two companies; if any part of the next couple of paragraphs is incorrect, I apologize sincerely to them, because I do not mean to misrepresent them at all.

NAQT regularly finishes tournaments in the couple of days before the first tournament using it occurs; in fact, this happens far more often than finishing the tournament. If I'm recalling correctly, they occasionally (very rarely) have to juggle their assigned sets so that they can get a set out the door to a host. This is fortunately a rare event, but it does happen.

HSAPQ has had similar issues with getting tournaments out the door and bumping against last chance promise dates.

Both companies have made calls for more writers; of course, quiz bowl being what it is, there's been more than a few cases of people saying they'd help, and then that help never happening. It is my impression that both companies are at the point where additional competent writers writing additional questions would be a boon, in terms of earlier completion dates and tournament quality.

The idea that we should divert resources from the production of these sets is, in a word, ludicrous. In more words, "absolutely insane," "an awful idea," "something that shouldn't be realistically considered," and "something that will seriously harm the spread of good quiz bowl." If we were to tell people to stop writing for IS sets, which are intended to appeal to hundreds of teams, and to instead work on IS+ sets, which are intended to appeal to tens of teams, quiz bowl would suffer.

If you read only one sentence in this post, please make it this one: If you care about quality high school quiz bowl tournaments, consisting of pyramidal tossups and well-written bonuses, becoming more popular, you must realize that making "regular difficulty tournaments" harder is the last thing you want to do; in fact, regular difficulty tournaments must become easier.

I'm not saying there can't be a couple of tournaments that are aimed at the best high school teams, but the fact is, the number of teams that those sets will have a positive effect on is less than the number of teams that would benefit from another well written regular season tournament. As such, it is absolutely incorrect to say that "I feel like it's unhelpful for everybody to try to produce a "regular" high school set" unless you're advocating making lesser teams answer even less questions than before, and having lower quality tournaments produced in general.
Attachments
all is96.xlsx
(17.12 KiB) Downloaded 263 times
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
Frater Taciturnus
Auron
Posts: 2463
Joined: Mon Dec 12, 2005 1:26 pm
Location: Richmond, VA

Re: "Regular" High School Difficulty

Post by Frater Taciturnus »

In case people can't handle the .xlsx attachment, I saved a .xls copy of Fred's spreadsheet.

Fred is absolutely right about this-- if the average team on this set can only get around 12 PPB then the set is being written harder than it needs to be. And this isnt something unique to this individual IS set-- I'm reasonably sure that if you took the same teams from IS-96 and had them play other "regular" difficulty sets, that similar results would ensue.
Attachments
all is96.xls
(40 KiB) Downloaded 227 times
Janet Berry
[email protected]
she/they
--------------
J. Sargeant Reynolds CC 2008, 2009, 2014
Virginia Commonwealth 2010, 2011, 2012, 2013,
Douglas Freeman 2005, 2006, 2007
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: "Regular" High School Difficulty

Post by Mechanical Beasts »

Frater Taciturnus wrote:In case people can't handle the .xlsx attachment, I saved a .xls copy of Fred's spreadsheet.

Fred is absolutely right about this-- if the average team on this set can only get around 12 PPB then the set is being written harder than it needs to be. And this isnt something unique to this individual IS set-- I'm reasonably sure that if you took the same teams from IS-96 and had them play other "regular" difficulty sets, that similar results would ensue.
Yeah, it's important to note that IS sets have probably the best penetration of any set--better than the run of the mill GSAC or LIST. It's a little like inferring that because main site GSAC has (for example) 14.6 average PPB it must have been of near-perfect difficulty: IS sets lack the massive geographic, experienced-team bias that most sets have in who ends up playing them.
Andrew Watkins
Nick
Wakka
Posts: 185
Joined: Sun Jul 29, 2007 4:41 pm

Re: "Regular" High School Difficulty

Post by Nick »

I cannot support Fred's post more strongly.

From the Middle School Collaboration up to ACF Nationals, this is the case.
Nick
User avatar
Boeing X-20, Please!
Rikku
Posts: 406
Joined: Sun Jul 26, 2009 2:40 pm
Location: Evanston, IL

Re: "Regular" High School Difficulty

Post by Boeing X-20, Please! »

Fred wrote: in the general sense, of tournaments intended to be normal difficulty for high school teams and the success of accomplishing that, these tournaments are too difficult (unless having the average ppb be well below 15 is acceptable, but I heavily disagree with that).
First, I'd like to note that I'm not saying that we change any and all sets to be harder, as your pointed out that set seems to be hard enough from that data, I'm rather advocating for an increase in the number of tournaments that are above "regular difficulty." HSAPQ writes some good question sets in many aspects including most importantly pyramidality, they seem to have a pretty good customer base, and their sets are widely well-received. I'm not arguing something along the lines of HSAPQ changing their difficulty to be harder than it is now: HSAPQ has good things going for it, it doesn't need to change. If you want to define "regular difficulty" by what HSAPQ is churning out, then I'm not in any way advocating for "regular difficulty" to be any harder. If you want to set 15 ppb as what regular difficulty aims for the average ppb to be, then yeah, that set you threw out was too hard for regular difficulty.

Second, I don't know if I would
Fred wrote: be willing to accept the assumption that IS-96 is representative, in the general sense, of tournaments intended to be normal difficulty for high school teams
I think this goes back to the issues brought up regarding classification of difficulty as discussed upthread. Most people would call NAQT IS-Sets and HSAPQ ACF-sets "regular difficulty," when in fact these two are pretty different. Since I don't have the nice nation-wide data that you have, I will work with what I have readily available to me, if HSAPQ has nation-wide data they'd like to share that goes against this conclusion, mea culpa for the selection bias.
There were two widely-attended tournaments on NAQT IS-sets in the Illinois circuit this year: Kickoffs (IS-98, I believe) and NAQT State (IS-104? In any case, it was not the set specifically written for state tournaments and therefore presumably of the same difficulty as that of IS-96) and one widely-attended tournament on HSAPQ: Loyburn (ACF-15). I chose HSAPQ and NAQT because they seem to be clearly defined as the standards in quizbowl and as Andy Watkins pointed out also have the greatest penetration/are played across the greatest breadth of areas. Unfortunately, Kickoffs used a wacky format with 5-person teams and more importantly bouncebacks that were not separated from actual bonus conversion so those stats are fairly useless.

Between these 2 tournaments, 11 teams attended in common. Loyburn was held in early November while State was held in late February, so presumably teams should be better due to being more experienced, being in the middle of a season instead of just starting up, and having more time to study. However, of these 11 teams only 2 of them had better points per bonus on the NAQT State set. One of those was Stevenson, who was missing Zach Blumenfeld at Loyburn and I am absolutely positive that due to his ability and place on that team he would have added more than .55 PPB to Stevenson's Loyburn PPB. The other was Greenville, whose PPB increased by just .49. The mean/median increase of PPB in teams that were better on NAQT than HSAPQ was .52, the mean and median decrease of PPB in teams that were better on HSAPQ than NAQT were 2.07 and 1.10. This seems to support that either NAQT IS-sets are above "regular difficulty" or HSAPQ sets are below "regular difficulty" in terms of hardness.


So basically, I agree with
George Berry wrote:Fred is absolutely right about this-- if the average team on this set can only get around 12 PPB then the set is being written harder than it needs to be.
but disagree with
George Berry wrote: And this isnt something unique to this individual IS set-- I'm reasonably sure that if you took the same teams from IS-96 and had them play other "regular" difficulty sets, that similar results would ensue.
because of the vast differences sets classified as being "regular difficulty" have in terms of their actual difficulty.

EDIT: Spreadsheet attached to have data in a more-viewable format.
IL HSAPQ-NAQT Comparison.xls
HSAPQ-NAQT Comparison
(9 KiB) Downloaded 256 times
Nolan Winkler
Loyola Academy '12
UChicago '16
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: "Regular" High School Difficulty

Post by AKKOLADE »

Secretary of Bobcats wrote:First, I'd like to note that I'm not saying that we change any and all sets to be harder, as your pointed out that set seems to be hard enough from that data, I'm rather advocating for an increase in the number of tournaments that are above "regular difficulty." HSAPQ writes some good question sets in many aspects including most importantly pyramidality, they seem to have a pretty good customer base, and their sets are widely well-received. I'm not arguing something along the lines of HSAPQ changing their difficulty to be harder than it is now: HSAPQ has good things going for it, it doesn't need to change.
I'm not opposed to additional sets being produced that are harder than regular difficulty if it doesn't negatively affect the production. That said, you'll have to look at what I wrote above about the production of questions for NAQT and HSAPQ - those are realities of the production of these sets. If infinite resources were available, I'd love to see a large number of harder sets regularly produced for the top teams; unfortunately, in reality, resources are less than infinite. I do think that it'd be good from an aesthetic stand point if a couple of sets were produced that aimed to fill this niche, though.

Regarding your method of comparison, I question if it's effective, given its small sample size and that of the eleven teams you looked at, five are currently in my top 100 and a sixth was in the 100 previously this year. But, what you put together would seem to indicate that you could expect the average ppb to be about 13.54 ppb (I got this value by adding the average ppb for IS-96 with what you calculated as the average difference in ppb between the two evaluated sets).

I ran the same analysis on HSAPQ 15 that I did on IS96. This included the following nine tournaments: BELLEROPHON, Danville, Fright, Quad States, Tiger Bowl, UCI CBCT, UNC Tar Heel, Trevor's Trivia, and VCU Fall. While not quite as diverse as the collection of areas as IS96, it does have a number of very strong and fairly weak fields.

I did not include Loyburn because of no ppb available.

166 teams played.

The average ppb is 13.33.
The median ppb is 12.43.
The third quartile ppb is 17.16.
The first quartile ppb is 9.07.

25.00-30.00: 1.81% (3)
20.00-24.99: 12.65% (21)
15.00-19.99: 23.49% (39)
10.00-14.99: 28.92% (48)
5.00-9.99: 30.12% (50)
0.00-4.99: 3.01% (5)

62.05% of teams averaged less than 15 ppb.

This is my interpretation of this data:
  • HSAPQ 15 is moderately easier than IS-96, as shown by the average and median being higher.
  • The field for HSAPQ 15 is pretty clearly top heavier than that of IS-96; this is reflected in the ~2 point increase in the third quartile value and in the ~1 point spread between HSAPQ 15's average & median ppb, where IS-96's average & median are close to each other.
  • If you accept 15 ppb as an acceptable hallmark of what an average team should have, HSAPQ 15 still falls short of this mark.
because of the vast differences sets classified as being "regular difficulty" have in terms of their actual difficulty.
I think you vastly overstate the differences in difficulty between HSAPQ and NAQT IS sets. The numbers I've run have lead me to believe that HSAPQ is indeed easier than NAQT. However, they also indicate that HSAPQ is still not close to having the average ppb be around 15 (though, they are closer) and could stand to be easier.

Edit: forgot to include the spreadsheet the first time.
Attachments
all ht15.xlsx
(15.29 KiB) Downloaded 223 times
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
jonah
Auron
Posts: 2383
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: "Regular" High School Difficulty

Post by jonah »

Fred wrote:I did not include Loyburn because of no ppb available.
Huh?
Jonah Greenthal
National Academic Quiz Tournaments
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: "Regular" High School Difficulty

Post by AKKOLADE »

I used the wrong link, I guess! I'll include it as soon as I get a chance.
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: "Regular" High School Difficulty

Post by grapesmoker »

If high school quizbowl is too easy for you, you are more than welcome at most collegiate tournaments. ACF in particular is certainly open to high school teams, so feel free to show up at the closest Regionals site to you in the coming year.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
Cheynem
Sin
Posts: 7220
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: "Regular" High School Difficulty

Post by Cheynem »

Yeah, Jerry makes an excellent point. High school teams have (and have frequently used) the option to "play up" and play most collegiate tournaments, a number of which to be honest (ACF Fall, MUT, Delta Burke) are not that much harder or are in fact easier than the hardest high school tournaments. Fred is absolutely right when he talks about resources. I would love to see a world in which there are a lot of question sets out there for teams to play on if they wish, but in a world where HSAPQ and NAQT struggle to produce what they do produce (and which frequently ends up being too hard for many teams anyway), that ain't gonna happen.

The last point I guess is semantics. If all of this is an an attempt to perhaps label HS tournaments different things, i.e. set up more of an easy/medium/hard label system (LIST as easy, GSAC as medium, HFT as hard, to think superficially), I'd be okay with that. I agree with Fred and friends though that the vast majority of HS tournaments should be easy to medium.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: "Regular" High School Difficulty

Post by AKKOLADE »

Updating the HSAPQ Tournament 16 numbers.

190 teams played.

The average ppb is 13.82.
The median ppb is 13.19.
The third quartile ppb is 17.82.
The first quartile ppb is 9.50.

25.00-30.00: 2.11% (4)
20.00-24.99: 15.26% (29)
15.00-19.99: 23.16% (44)
10.00-14.99: 29.47% (56)
5.00-9.99: 27.37% (52)
0.00-4.99: 2.63% (5)

59.47% of teams averaged less than 15 ppb.

Attached is the updated spreadsheet.

This doesn't change the rest of my post above.
Attachments
all ht15 corrected.xlsx
(16.11 KiB) Downloaded 240 times
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: "Regular" High School Difficulty

Post by theMoMA »

Here is my attempt to wade into this morass.

Fred has outlined the reasons why creating tournaments at a higher-than-regular level for high school use is not an efficient use of time. I can't agree with him enough. As someone who's been on the ground working on regular high school tournaments for the better part of five years, I can tell you that the production teams are stretched thin.

The problem that you see with so-called "regular" tournaments having a wide range of difficulty has everything to do with the notorious inability of high schoolers to gauge appropriate difficulty and nothing to do with an ontological problem in the concept of regular difficulty. High schoolers also tend to want it both ways: to write on whatever they want and to have their tournament mirrored in as many places as possible. This suggests to me that high schoolers writing tournaments have to figure out a better way to limit themselves to more appropriate answers before they write an ostensible regular-difficulty tournament.

Some might counter "why not just call those harder high school tournaments 'regular-plus' and mirror them for a more skilled target audience?" First off, I'm not convinced, based on the high school sets I've seen, that what high schoolers think is hard-but-appropriate actually provides for fair competition, let alone fairer competition than easier questions. Secondly, and foremost, this creates a social problem. Just like no one wants to write the easy question at Chicago Open, no high school team will want to write the regular-difficulty tournament when the option to create a shiny nationals-level competition is out there. Let's be real here: there are only a handful of teams capable of handling regular-plus, and there are scant regions in which there is a tournament-field-worth of such teams. The tournaments are already too hard for their target audiences. Why in the world would we institutionalize that overdifficulty?
Andrew Hart
Minnesota alum
User avatar
Stained Diviner
Auron
Posts: 5085
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland
Contact:

Re: "Regular" High School Difficulty

Post by Stained Diviner »

Obviously, there are two problems here. One is that regular high school difficulty has become a useless label, and the other is that it is difficult finding writers for what properly should be called regular high school difficulty.

The first problem can be fixed by realizing that sets like HFT, GSAC, and BHSAT are significantly different from what we want regular high school difficulty to be like, so they should be clearly labeled as more difficult sets. These are very good sets, and they have an audience that likes them, and teams learn important information by playing them. Also, very good teams get more than 20 PPB on them and often buzz in during the first half of the question, and pretty good teams get more than 15 PPB and often buzz in on late middle clues. I don't think we want to eliminate these sets, but we should expect them to label themselves honestly, so that when LIST honestly calls itself regular difficulty teams don't show up (and TDs don't purchase questions) expecting a tournament like GSAC. Many of these sets are written to cater to the field that shows up at the site where they are being written, and it's perfectly reasonable that this happens. Problems occur when mirrors with significantly different field strengths use the same questions without expecting different results. Also, TDs with average or weak fields sometimes ignore the possibility of using an independent set because independent sets tend to be difficult, and it is often difficult to gauge the difficulty level of independent sets without actually using them (especially because sometimes the same independent set varies in difficulty from year to year).

The second problem is that we don't have enough people willing to write regular difficulty. Because of this, NAQT and HSAPQ have trouble completing their sets early enough to have several editors fine tune them, HSAPQ recently cut back on the number of sets it produces each year, and requests to NAQT to increase production can't be taken seriously even if there is a market for more questions. Furthermore, head editors at those companies have to constantly remind their writers and editors to write easier questions, and even then the questions end up a little too difficult. Frankly, I wish I knew how to solve this problem. More people should write more questions for NAQT and HSAPQ, but that's easier said than done.
David Reinstein
Head Writer and Editor for Scobol Solo, Masonics, and IESA; TD for Scobol Solo and Reinstein Varsity; IHSSBCA Board Member; IHSSBCA Chair (2004-2014); PACE President (2016-2018)
jonah
Auron
Posts: 2383
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: "Regular" High School Difficulty

Post by jonah »

Westwon wrote:Frankly, I wish I knew how to solve this problem. More people should write more questions for NAQT and HSAPQ, but that's easier said than done.
One fact that I think gets overlooked a lot is that NAQT allows high school coaches to write for collegiate and middle school sets. (I imagine if there were interest, it would also allow middle school coaches to write for high school and collegiate sets, but as far as I know that hasn't come up.) Currently no one is taking advantage of this, but it would be great if people would, since that would allow other writers to focus resources more on high school sets. (Reinstein, I'm not aiming this suggestion at you particularly, though I'm sure your contributions would be welcome.)
Jonah Greenthal
National Academic Quiz Tournaments
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: "Regular" High School Difficulty

Post by AKKOLADE »

I'll just point out real quick that GSAC wasn't very much harder than an IS set for top teams. Granted, I haven't looked at teams that aren't in consideration for the rankings, so it could be a different story with them, but I did want to point out that while in past years GSAC has been more difficult, this year's iteration is not of the same difficulty.
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
Adventure Temple Trail
Auron
Posts: 2754
Joined: Tue Jul 15, 2008 9:52 pm

Re: "Regular" High School Difficulty

Post by Adventure Temple Trail »

Westwon wrote:... BHSAT [is] significantly different from what we want regular high school difficulty to be like, so [it] should be clearly labeled as more difficult
I hate to derail here, but is there statistical evidence to back up this claim? We tried pretty hard this year to ensure that BHSAT hewed very closely to accessible high school difficulty for all teams that played it, and as far as I know we kept new, difficult clues close to the very beginnings of tossups, as is the case with basically every high school set. We didn't label the set as "more difficult" because, to our knowledge, it wasn't "more difficult" in a statistically significant sense of the term. No teams, including local teams, got below 10 PPB at the main site, and I see only three that have gotten below 10 ppb so far out of three mirrors which have happened thus far, with a total of 4 out of 39 teams at or below 10 PPB. (I find the comparison to HFT, which has been very difficult for five years running and saw 20 out of 36 teams got at or below 10 PPB in the prelims in its most recent incarnation, to be particularly jarring, bordering on offensive.) Since the set is not clear, if you have any specific concerns about difficulty outliers in the set, feel free to PM or email me. As of now, a month after the first use of the set, I've received some emails detailing factual mistakes (which should all be corrected), but none about particular difficulty outliers.

As to other issues in this thread:
-If your set is far above the difficulty of an IS or HSAPQ set, it is your duty as a tournament writer to announce this to teams beforehand. (I'm looking at you, HFT.)
-If you are good at high school quizbowl (i.e. getting over 20 PPB at every tournament), step back from the computer for a second, pat yourself on the back for your accomplishment, and realize that high school quizbowl is not too easy for you (or the nation) just because you are good at it; unless you are effectively powering every tossup, 30ing every bonus, and winning every game, there is still room for improvement that you haven't done yet at the difficulty level which you're playing.

-The writing of sets accessible to everyone should precede the writing of sets which are aimed at the much, much smaller audience that wants more difficult fare. This goes for high school and college.
Matt Jackson
University of Chicago '24
Yale '14, Georgetown Day School '10
member emeritus, ACF
jonah
Auron
Posts: 2383
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: "Regular" High School Difficulty

Post by jonah »

I don't have data, but I agree with Matt that this year's BHSAT was approximately regular difficulty.
Jonah Greenthal
National Academic Quiz Tournaments
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: "Regular" High School Difficulty

Post by AKKOLADE »

I slapped three more together while taking a break from work.

IS-98: 223 teams played.

The average ppb is 12.34.
The median ppb is 11.45.
The third quartile ppb is 16.11.
The first quartile ppb is 8.81.

IS-100: 218 teams played.

The average ppb is 11.98.
The median ppb is 11.68.
The third quartile ppb is 15.35.
The first quartile ppb is 8.27.

IS-102: 191 teams played.

The average ppb is 12.72.
The median ppb is 12.25.
The third quartile ppb is 16.61.
The first quartile ppb is 8.68.
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
Cheynem
Sin
Posts: 7220
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: "Regular" High School Difficulty

Post by Cheynem »

BHSAT at our site saw somewhat suppressed stats because three teams were very inexperienced, not very good, or both. Neither BHSAT or GSAC seemed as hard as HFT.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
Stained Diviner
Auron
Posts: 5085
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland
Contact:

Re: "Regular" High School Difficulty

Post by Stained Diviner »

My impression of BHSAT when listening to it was that it could have been easier, but a quick perusal of stats backs up Matt's claim that it played at a similar difficulty to IS, so I was wrong there, and that derailment is my fault. I still am on the fence with GSAC. Here are the stats from the bottom division of our tournament, which is not a good sample because it consists of teams that self-select as not elite. That shows that the set should have been easier, but Fred's statistics show that IS also should have been easier, so it's hard to tell the difference at this point.

Maybe we need to stop labeling Fall Novice and SCOP as Novice Sets. Call them average difficulty and encourage the production of more sets like them, with the understanding that there is also a market for sets not like them.

Also, I hope people don't gloss over Jerry's point too quickly, because he's right.
David Reinstein
Head Writer and Editor for Scobol Solo, Masonics, and IESA; TD for Scobol Solo and Reinstein Varsity; IHSSBCA Board Member; IHSSBCA Chair (2004-2014); PACE President (2016-2018)
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: "Regular" High School Difficulty

Post by AKKOLADE »

Okay, I've finished looking at the NAQT numbers.

IS-98: 223 teams played.

The average ppb is 12.34.
The median ppb is 11.45.
The third quartile ppb is 16.11.
The first quartile ppb is 8.81.

25.00-30.00: 0% (0)
20.00-24.99: 8.97% (20)
15.00-19.99: 19.73% (44)
10.00-14.99: 34.53% (77)
5.00-9.99: 33.63% (75)
0.00-4.99: 3.14% (7)

71.30% of teams averaged less than 15 ppb.

IS-100: 218 teams played.

The average ppb is 11.98.
The median ppb is 11.68.
The third quartile ppb is 15.35.
The first quartile ppb is 8.27.

25.00-30.00: 0% (0)
20.00-24.99: 4.11% (9)
15.00-19.99: 24.20% (53)
10.00-14.99: 34.70% (76)
5.00-9.99: 32.42% (71)
0.00-4.99: 4.11% (9)

71.23% of teams averaged less than 15 ppb.

IS-102: 191 teams played.

The average ppb is 12.72.
The median ppb is 12.25.
The third quartile ppb is 16.61.
The first quartile ppb is 8.68.

25.00-30.00: 0% (0)
20.00-24.99: 10.47% (20)
15.00-19.99: 24.08% (46)
10.00-14.99: 29.84% (57)
5.00-9.99: 31.94% (61)
0.00-4.99: 3.66% (7)

65.45% of teams averaged less than 15 ppb.

IS-104: 179 teams played.

The average ppb is 13.62.
The median ppb is 13.80.
The third quartile ppb is 17.83.
The first quartile ppb is 9.63.

25.00-30.00: 0% (0)
20.00-24.99: 11.17% (20)
15.00-19.99: 29.05% (52)
10.00-14.99: 34.08% (61)
5.00-9.99: 21.79% (39)
0.00-4.99: 3.91% (7)

59.78% of teams averaged less than 15 ppb.

IS-105: 279 teams played.

The average ppb is 13.94.
The median ppb is 13.77.
The third quartile ppb is 16.90.
The first quartile ppb is 10.88.

25.00-30.00: 0.36% (1)
20.00-24.99: 9.68% (27)
15.00-19.99: 31.54% (88)
10.00-14.99: 39.07% (109)
5.00-9.99: 19.00% (53)
0.00-4.99: 0.36% (1)

58.42% of teams averaged less than 15 ppb.

Overall IS sets - 1361 stat lines.

The average ppb is 12.75.
The median ppb is 12.46.
The third quartile ppb is 16.43.
The first quartile ppb is 9.00.

25.00-30.00: 0.15% (2)
20.00-24.99: 8.37% (114)
15.00-19.99: 24.54% (334)
10.00-14.99: 35.19% (479)
5.00-9.99: 27.99% (381)
0.00-4.99: 3.75% (51)

66.94% of teams averaged less than 15 ppb.

The range of 1 standard deviation from the mean is 7.82-17.69 ppb
The range of 2 standard deviations from the mean is 2.88-22.63 ppb
Attachments
all is.xlsx
(90.96 KiB) Downloaded 219 times
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: "Regular" High School Difficulty

Post by Sen. Estes Kefauver (D-TN) »

I want to focus on another aspect of this argument that I find to be particularly pernicious, which is that easier sets are not "real enough" to distinguish between high end teams, which was one of William's criticisms of LIST. I want to instead posit to you all that as long as a set uses a fair distribution, has a tossup length that is IS-set length or greater, and has good answers and a proper selection of clues, then that is sufficient to provide distinction between teams (unless the teams are themselves equally good, in which case it's not like you could really find better sets to distinguish them). All it does is move the goalposts from "which team can get more tossups in the middle, with a few good leadin buzzes thrown in, and can get a lot of 20s and a decent number of 30s," which is how a lot of these games go at the national level, to "which team can get the most powers and knows enough to confidently buzz in on leadins tossup after tossup, while also 30ing most of their bonuses." These games will be higher scoring, and they will perhaps frustrate the loser if there are some good buzzer races lost, but buzzer races happen in all levels of the game, and these sets will still provide enough of a chance for good teams to sufficiently be distinguished.

We see a good example of this at the LIST mirror in Georgia, which supposedly was not sufficient to distinguish between good teams. Alpharetta got in the top 20 at PACE last year, returns their scoring, and remains somewhere in the top 20. The Centennial hybrid team was based around a team that is also in the top 20, somewhere above Alpharetta, probably in the top 10 of the country. The Centennial hybrid team went 10-0 for the day, had a points per game average of 582, and had the tossup line of 114 powers, 26 10s, and 22 -5s. Alpharetta had an 8-2 record for the day, losing twice to Centennial, 492 points per game, and a tossup line of 82 powers, 43 10s, and 28 -5s. In the first match between the two, Centennial beat Alpharetta 450-335. Centennial got 9 powers, 2 10s, and 3 -5s. Alpharetta got 7 powers, 2 10s, and 2 -5s. In the second game, Centennial beat Alpharetta 470-290. Centennial got 7 powers, 5 10s, and 3 -5s. Alpharetta got 3 powers, 5 10s, and 3 -5s. Unfortunately all the bonus data is incorrect for these teams, but I also find it hard to believe Centennial was not pushing 30 ppb, while Alpharetta was undoubtedly doing something a little less than that. In any case, from all this data, it is very clear to me that, when distinguishing between these two highly ranked teams, rather than muddying the data, the LIST set allowed Centennial the chance to outperform Alpharetta both throughout the day and in their two matches in a way that left them undeniably the deserving champions of their site, and left Alpharetta undeniably the second best team there. This means from the perspective of providing fair games to rank top teams, LIST did its job, without sacrificing ease or accessibility to the bottom of the field, contrary to what William argued. The situation with Centennial vs. Alpharetta seems to be comparable to a lot of other fields out there - you are almost never going to find tournaments with 8 of the top 20 in attendance outside of nationals season - so this seems to me to be a completely legitimate way to rank good teams at most tournaments.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
User avatar
kayli
Auron
Posts: 1525
Joined: Sat Jun 21, 2008 10:15 pm
Location: Illinois

Re: "Regular" High School Difficulty

Post by kayli »

Jeremy Gibbs Freesy Does It wrote:I want to focus on another aspect of this argument that I find to be particularly pernicious, which is that easier sets are not "real enough" to distinguish between high end teams, which was one of William's criticisms of LIST.
I disagree with this completely. At a certain level of competition, easy sets are much worse at determining who is better because two good teams essentially starting playing of the first 3 or so clues or tossups which reduces basically to buzzer races. Then there are issues with bonuses which may become too easy so the difference becomes those one or two things you didn't thirty or that one bonus on Canadian indie bands that you bageled. There's a reason why we use higher difficulty sets to differentiate between the best teams, and that's because they're better at doing just that. Else, there would be no need for higher difficulty sets to be used at nationals other than the novelty of them.
Kay, Chicago.
User avatar
Auroni
Auron
Posts: 3145
Joined: Thu Nov 15, 2007 6:23 pm

Re: "Regular" High School Difficulty

Post by Auroni »

Ar$oni$t$ Get All the Girl$ wrote:
Jeremy Gibbs Freesy Does It wrote:I want to focus on another aspect of this argument that I find to be particularly pernicious, which is that easier sets are not "real enough" to distinguish between high end teams, which was one of William's criticisms of LIST.
I disagree with this completely. At a certain level of competition, easy sets are much worse at determining who is better because two good teams essentially starting playing of the first 3 or so clues or tossups which reduces basically to buzzer races.
This is not actually what happens. I'd urge you to observe matches between top teams on easier sets before you make posts like this.
Auroni Gupta (she/her)
User avatar
kayli
Auron
Posts: 1525
Joined: Sat Jun 21, 2008 10:15 pm
Location: Illinois

Re: "Regular" High School Difficulty

Post by kayli »

every time i refresh i have a new name wrote:
Ar$oni$t$ Get All the Girl$ wrote:
Jeremy Gibbs Freesy Does It wrote:I want to focus on another aspect of this argument that I find to be particularly pernicious, which is that easier sets are not "real enough" to distinguish between high end teams, which was one of William's criticisms of LIST.
I disagree with this completely. At a certain level of competition, easy sets are much worse at determining who is better because two good teams essentially starting playing of the first 3 or so clues or tossups which reduces basically to buzzer races.
This is not actually what happens. I'd urge you to observe matches between top teams on easier sets before you make posts like this.
It's a bit exaggerated, but when you get a hundred of powers in a tournament, it's not too far off.
Kay, Chicago.
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: "Regular" High School Difficulty

Post by grapesmoker »

Ar$oni$t$ Get All the Girl$ wrote:It's a bit exaggerated, but when you get a hundred of powers in a tournament, it's not too far off.
I hate wading into high school discussions because it's annoying to have to post long reams of evidence for the sole purpose of disabusing people of absurd opinions that they hold by virtue of their inexperience, but:

You are simply wrong. There's absolutely no reason to believe that the LIST set (which I have not seen and am not involved with in any way shape or form) did not properly differentiate the two top teams at this tournament. Charlie gave you an exhaustive breakdown of how it did so, based on the statistics of those teams. I know quizbowl has difficulty being convinced by such flimsy notions as "facts," but you would do well to at least pay lip service to his effort rather than just posting "NUH-UH."

Here's a thing, you guys. There are people posting here who have been involved in quizbowl for many years; there are also people who have been involved in quizbowl for only a year or two but think that on the basis of some decent performances at some local events they already know everything. I appreciate the desire to play up a level (as a noted purveyor of hard tournaments, this is just fine with me), but I also appreciate the fact that people who work at the high school level on a regular basis (that would be folks like Fred, Charlie, and Matt Weiner) have a global view that you, Kay Li and William from Alpharetta, do not possess. So maybe, just maybe, when they tell you that a) the numbers across the board do not support the position that high school quizbowl is too easy, and b) actually show you how those numbers separate even the top teams, you should pay attention. Posting on hsqb about how awesome you are and how everything is totally easy will not make you better and it also won't do anything to help out those teams for whom it is not, in fact, totally easy. If you're so great, that's awesome, come play some ACF tournaments, just so long as you stop this inane debate in which you have no leg to stand on.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: "Regular" High School Difficulty

Post by Sen. Estes Kefauver (D-TN) »

At a certain level of competition, easy sets are much worse at determining who is better because two good teams essentially starting playing of the first 3 or so clues or tossups which reduces basically to buzzer races.
I would appreciate your opinion on this issue a tad more if you ever actually were on a top 20 team that was playing another top 20 team in a tournament, which is laughably far from the case.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
User avatar
Steeve Ho You Fat
Auron
Posts: 1138
Joined: Mon Jun 01, 2009 11:48 pm

Re: "Regular" High School Difficulty

Post by Steeve Ho You Fat »

Ar$oni$t$ Get All the Girl$ wrote:
every time i refresh i have a new name wrote:
Ar$oni$t$ Get All the Girl$ wrote:
Jeremy Gibbs Freesy Does It wrote:I want to focus on another aspect of this argument that I find to be particularly pernicious, which is that easier sets are not "real enough" to distinguish between high end teams, which was one of William's criticisms of LIST.
I disagree with this completely. At a certain level of competition, easy sets are much worse at determining who is better because two good teams essentially starting playing of the first 3 or so clues or tossups which reduces basically to buzzer races.
This is not actually what happens. I'd urge you to observe matches between top teams on easier sets before you make posts like this.
It's a bit exaggerated, but when you get a hundred of powers in a tournament, it's not too far off.
I can see both sides of this: on an Ohio A set tournament that had the top two teams in Ohio according to our polls and DCC, it just turned into a massive buzzer-race set, with lots of upsets because every other question was decided on speed as both teams would know the first one or two clues. However, saying that just because two teams had lots of powers for a tournament means that when they play each other the results will be based on speed isn't true. For example, at Ohio's BATE mirror we used a power-marked version of the set that was extraordinarily generous in its giving of powers. Northmont, who won, got 98, Copley got 99, Olmsted Falls got 87, and us, who finished fourth, got 48. Because Northmont, Copley, and Olmsted Falls got a very similar number of powers, does that mean the set was unable to differentiate between them? Not at all. I thought it did this perfectly well. While this tournament may have had unusually generous power-marking, and I don't know whether or not LIST did, I think my point stands that there can be multiple clues in power, which can be arranged pyramidally and have difficult lead-ins, thereby differentiating teams when Centennial is powering something on the first clue and Alpharetta on the second.
Joe Nutter
PACE Emeritus
Michigan State University '14
Walnut Hills High School '11
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: "Regular" High School Difficulty

Post by Sen. Estes Kefauver (D-TN) »

I can see both sides of this: on an Ohio A set tournament that had the top two teams in Ohio according to our polls and DCC, it just turned into a massive buzzer-race set, with lots of upsets because every other question was decided on speed as both teams would know the first one or two clues.
I will just note that the fundamental problem here is that A-sets are 3 lines long. I'm not opposed to using A-sets, especially in fields with lots of inexperienced teams, but there are real limitations on the ability of a set to distinguish between good teams when the tossups are shorter than usual. If you were to add another 2 lines to every tossup to a currently existing A-set, you would basically have a very playable, low difficulty set that will still distinguish between great teams.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
User avatar
ryandillon
Wakka
Posts: 151
Joined: Sun Jul 12, 2009 10:47 am
Location: Detroit

Re: "Regular" High School Difficulty

Post by ryandillon »

Jeremy Gibbs Freesy Does It wrote: I will just note that the fundamental problem here is that A-sets are 3 lines long. I'm not opposed to using A-sets, especially in fields with lots of inexperienced teams, but there are real limitations on the ability of a set to distinguish between good teams when the tossups are shorter than usual. If you were to add another 2 lines to every tossup to a currently existing A-set, you would basically have a very playable, low difficulty set that will still distinguish between great teams.
Yeah that's pretty much it. A sets are what they are, but when you're trying to differentiate between teams, you need a little bit more length. If a six or seven line tossup is well-written, there's no reason, no matter what the answer choice is, that it can't distinguish between two good teams. And I also don't really think that the presence of "buzzer races" really makes a set good or bad at distinguishing between teams. Buzzer races happen, especially when two good teams show up to play. Yeah there definitely is a point where they become excessive, but a buzzer race here or there is part of the game.
Ryan Dillon

Detroit Catholic Central Class of 2011
University of Michigan Class of 2015
User avatar
Steeve Ho You Fat
Auron
Posts: 1138
Joined: Mon Jun 01, 2009 11:48 pm

Re: "Regular" High School Difficulty

Post by Steeve Ho You Fat »

Jeremy Gibbs Freesy Does It wrote:
I can see both sides of this: on an Ohio A set tournament that had the top two teams in Ohio according to our polls and DCC, it just turned into a massive buzzer-race set, with lots of upsets because every other question was decided on speed as both teams would know the first one or two clues.
I will just note that the fundamental problem here is that A-sets are 3 lines long. I'm not opposed to using A-sets, especially in fields with lots of inexperienced teams, but there are real limitations on the ability of a set to distinguish between good teams when the tossups are shorter than usual. If you were to add another 2 lines to every tossup to a currently existing A-set, you would basically have a very playable, low difficulty set that will still distinguish between great teams.
Yeah, I intended to say basically that but I guess I didn't make it clear. There's absolutely nothing wrong with having easier tossups on easy answers that also have challenging early clues, thereby differentiating better teams while minimizing dead questions among newer teams. Sadly, it seems like there's only a choice between A set buzzer races and things like HFT, which have some tossups that are really impossible for non-top teams.
Joe Nutter
PACE Emeritus
Michigan State University '14
Walnut Hills High School '11
User avatar
kayli
Auron
Posts: 1525
Joined: Sat Jun 21, 2008 10:15 pm
Location: Illinois

Re: "Regular" High School Difficulty

Post by kayli »

grapesmoker wrote:
Ar$oni$t$ Get All the Girl$ wrote:It's a bit exaggerated, but when you get a hundred of powers in a tournament, it's not too far off.
I hate wading into high school discussions because it's annoying to have to post long reams of evidence for the sole purpose of disabusing people of absurd opinions that they hold by virtue of their inexperience, but:

You are simply wrong. There's absolutely no reason to believe that the LIST set (which I have not seen and am not involved with in any way shape or form) did not properly differentiate the two top teams at this tournament. Charlie gave you an exhaustive breakdown of how it did so, based on the statistics of those teams. I know quizbowl has difficulty being convinced by such flimsy notions as "facts," but you would do well to at least pay lip service to his effort rather than just posting "NUH-UH."

Here's a thing, you guys. There are people posting here who have been involved in quizbowl for many years; there are also people who have been involved in quizbowl for only a year or two but think that on the basis of some decent performances at some local events they already know everything. I appreciate the desire to play up a level (as a noted purveyor of hard tournaments, this is just fine with me), but I also appreciate the fact that people who work at the high school level on a regular basis (that would be folks like Fred, Charlie, and Matt Weiner) have a global view that you, Kay Li and William from Alpharetta, do not possess. So maybe, just maybe, when they tell you that a) the numbers across the board do not support the position that high school quizbowl is too easy, and b) actually show you how those numbers separate even the top teams, you should pay attention. Posting on hsqb about how awesome you are and how everything is totally easy will not make you better and it also won't do anything to help out those teams for whom it is not, in fact, totally easy. If you're so great, that's awesome, come play some ACF tournaments, just so long as you stop this inane debate in which you have no leg to stand on.
This is the most absurd argument ad hominem blathering I've seen in a while. You're arguing against points that no one has said. Speaking for myself, I'm not propping up my own team or proclaiming my own superiority to the heavens, and nothing I've said has done that. I know better than anyone how mediocre Pensacola is. Also, I've never said that tournaments are too easy. I'm just saying that easy tournaments are worse at differentiating teams based on knowledge than harder tournaments. When you have teams averaging over 8-10 powers a round, there's obviously a lack in depth of tossup difficulty. That was my evidence. I think Charlie's example suffers from a couple issues including a small sample size and arguable comparability in teams which makes the results not very significant. It'd be nice to see an counterargument instead of blathering on about how great certain people have been for quizbowl and how people should stop questioning them.
Jeremy Gibbs Freesy Does It wrote:
At a certain level of competition, easy sets are much worse at determining who is better because two good teams essentially starting playing of the first 3 or so clues or tossups which reduces basically to buzzer races.
I would appreciate your opinion on this issue a tad more if you ever actually were on a top 20 team that was playing another top 20 team in a tournament, which is laughably far from the case.
Your example doesn't give good enough evidence because there's a small sample size (one instance!) and I don't think Alpharetta and Chatahoochee were comparable in skill anyway. Chatahoochee has beaten Alpharetta in pretty much every meeting from what a quick search gives me.

Also, I forgot that I was supposed to be a part of a top 20 team to have an opinion. Oh wait, that's clearly fallacious. The fact that I'm not a top competitor does not automatically invalidate my opinion.

User was banned for three days for failing to follow the basic rules of logical argumentation and metaposting. - Mgmt.
Kay, Chicago.
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: "Regular" High School Difficulty

Post by Sen. Estes Kefauver (D-TN) »

Kay Li, you are a huge idiot! I can't believe you still post here because everybody wants you to go away!

User was banned for three days for a personal attack - Mgmt.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
User avatar
kayli
Auron
Posts: 1525
Joined: Sat Jun 21, 2008 10:15 pm
Location: Illinois

Re: "Regular" High School Difficulty

Post by kayli »

Jeremy Gibbs Freesy Does It wrote:
I can see both sides of this: on an Ohio A set tournament that had the top two teams in Ohio according to our polls and DCC, it just turned into a massive buzzer-race set, with lots of upsets because every other question was decided on speed as both teams would know the first one or two clues.
I will just note that the fundamental problem here is that A-sets are 3 lines long. I'm not opposed to using A-sets, especially in fields with lots of inexperienced teams, but there are real limitations on the ability of a set to distinguish between good teams when the tossups are shorter than usual. If you were to add another 2 lines to every tossup to a currently existing A-set, you would basically have a very playable, low difficulty set that will still distinguish between great teams.
I can see sorta this. But I still think it's less optimal unless the first clues were quite robust in difficult-ish clue selection.
Kay, Chicago.
User avatar
Adventure Temple Trail
Auron
Posts: 2754
Joined: Tue Jul 15, 2008 9:52 pm

Re: "Regular" High School Difficulty

Post by Adventure Temple Trail »

Ar$oni$t$ Get All the Girl$ wrote:When you have teams averaging over 8-10 powers a round, there's obviously a lack in depth of tossup difficulty.
I don't believe this particular claim to be true. In most high school sets with power marks, there are between 2 and 4 lines of information within power, which usually translates to between 3 to 8 distinct clues on which top teams can either buzz if they know the answer, guess, or not buzz depending on their level of knowledge/skill. If powers only lasted through the first clue of a question, I might buy the idea that a high power rate among several teams playing the set means shallow questions, but as it stands I don't -- there are enough clues in most pyramidal sets that teams still beat each other to tossups without a buzzer race occurring the overwhelming majority of the time. And if teams are averaging over 8-10 powers around, it just means the process of buzzing in before the other team is happening earlier in many tossups - not necessarily at the same time in every tossup as every other team with a similar number of powers per game.
Matt Jackson
University of Chicago '24
Yale '14, Georgetown Day School '10
member emeritus, ACF
User avatar
Auroni
Auron
Posts: 3145
Joined: Thu Nov 15, 2007 6:23 pm

Re: "Regular" High School Difficulty

Post by Auroni »

Look dude, you have never seen two top teams play a regular difficulty set. I have. What you intimated happens in such a match does not actually happen. Stop going with your gut and instead defer to the people who have first-hand experience, such as Charlie, Matt, and Fred do, of what happens when a wide variety of teams play on regular high school difficulty sets.
Auroni Gupta (she/her)
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: "Regular" High School Difficulty

Post by grapesmoker »

Ar$oni$t$ Get All the Girl$ wrote:This is the most absurd argument ad hominem blathering I've seen in a while.
Words mean things and that's not what the term "ad hominem" means.
You're arguing against points that no one has said. Speaking for myself, I'm not propping up my own team or proclaiming my own superiority to the heavens, and nothing I've said has done that. I know better than anyone how mediocre Pensacola is. Also, I've never said that tournaments are too easy. I'm just saying that easy tournaments are worse at differentiating teams based on knowledge than harder tournaments. When you have teams averaging over 8-10 powers a round, there's obviously a lack in depth of tossup difficulty. That was my evidence.
You have no evidence. You just proclaimed your conclusion despite the fact that it doesn't follow from your premises. There is not "obviously" any lack of tossup difficulty and your claim is unfounded.
I think Charlie's example suffers from a couple issues including a small sample size and arguable comparability in teams which makes the results not very significant. It'd be nice to see an counterargument instead of blathering on about how great certain people have been for quizbowl and how people should stop questioning them.
I did not say "don't question these people." I am telling you that, unlike you, these people have a genuine claim to expertise in writing quizbowl questions for high school audiences. Therefore, their claims should be taken seriously, particularly when, like Fred and Charlie, they've put in some effort towards showing you the statistics that back up their claims. You haven't shown anything other than the fact that you looked at a set of results from a tournament once.
Also, I forgot that I was supposed to be a part of a top 20 team to have an opinion. Oh wait, that's clearly fallacious. The fact that I'm not a top competitor does not automatically invalidate my opinion.
What invalidates your opinion is its wrongness.

Whatever, dude. I've come in here to say my thing because I think this whole discussion has taken on a pretty insane tone, thanks in large part to you and your ridiculous claims and your terrible non-logic. You are welcome to continue the crusade to make high school quizbowl more difficult by furiously posting every 15 seconds about how easy it is, and I am just going to assume that people who actually know something are just going to ignore you, as they should.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
Tanay
Rikku
Posts: 427
Joined: Wed Jun 03, 2009 5:05 pm

Re: "Regular" High School Difficulty

Post by Tanay »

Ar$oni$t$ Get All the Girl$ wrote:When you have teams averaging over 8-10 powers a round, there's obviously a lack in depth of tossup difficulty.
Not to beat a dead horse, but a look at the 2010 NSC stats here suggests that good teams were getting 8-10 powers per round with some degree of consistency. I don't think this is indicative of a "lack in depth of tossup difficulty" or anything of the like.
Tanay
ex-Berkeley, ex-Bellarmine
User avatar
kayli
Auron
Posts: 1525
Joined: Sat Jun 21, 2008 10:15 pm
Location: Illinois

Re: "Regular" High School Difficulty

Post by kayli »

Whatever, dude. I've come in here to say my thing because I think this whole discussion has taken on a pretty insane tone, thanks in large part to you and your ridiculous claims and your terrible non-logic. You are welcome to continue the crusade to make high school quizbowl more difficult by furiously posting every 15 seconds about how easy it is, and I am just going to assume that people who actually know something are just going to ignore you, as they should.
When have I ever done this? Honestly, all I've said is that easier question sets do worse at differentiating teams based on skill. I've never said that quizbowl is too easy, and I'm not on a crusade to make it harder. The gist of my argument is as follows: Teams are getting ridiculous amounts of powers. Powers are on the hardest parts of tossups and come earliest. More powers by teams indicates that they can get the hardest parts easier and earlier. If a set has too many powers, that means that the best teams are getting them earlier and easier. If teams are getting them earlier and easier, that usually indicates that, when two good teams face each other, they're fighting for the hardest and earliest clues. Thus, I think that if there are a large number of powers on a set, that doesn't do as well at differentiation teams as a harder set which operates with more room to differentiate them.

I'll admit that I'm just extrapolating from what happens between two teams of okay quality compete on an A-set, but I still think that harder sets do better at differentiating the knowledge of two teams. That's why we use them in state and national tournaments after all. They give more room for differentiation while easier sets have less room.

It looks like from Tanay and Matt's posts that number of powers shouldn't be used to indicate the easiness of a set. I can accept that. So I guess I'll revise my opinion to: easy tournaments are less optimal at differentiation between top teams as harder tournament, bbut easy tournaments can still do it.
Kay, Chicago.
Andrew's a Freshman
Wakka
Posts: 102
Joined: Sun Mar 02, 2008 11:12 am
Location: Norcross, GA

Re: "Regular" High School Difficulty

Post by Andrew's a Freshman »

grapesmoker wrote:this whole discussion has taken on a pretty insane tone
Yeah. Which, in my opinion, is taking away from a viable point that was at one point being discussed. Question writers should put in a good deal of effort in describing their sets accurately. This is not to say that this is not being done already. However, I think it may be more important than some people realize. As has been mentioned, there is about a two-fold variation in question difficulty that's acceptable in high school. A set that is thought to be "slightly above regular difficulty" is going to be hard for teams without much experience even if the information doesn't seem all that unreasonable. Really, "regular high school difficulty" is hard for a good percentage of teams as stats have shown.

Fred compiled a great deal of NAQT statistics as well as relative conversion rates for different sets in relation to his rankings. Teams should be aware that 70-80% of teams will likely have under 15 PPB on a given set before signing up.
Norcross High School '11
Northwestern University '15
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: "Regular" High School Difficulty

Post by grapesmoker »

Ar$oni$t$ Get All the Girl$ wrote:The gist of my argument is as follows: Teams are getting ridiculous amounts of powers. Powers are on the hardest parts of tossups and come earliest. More powers by teams indicates that they can get the hardest parts easier and earlier. If a set has too many powers, that means that the best teams are getting them earlier and easier. If teams are getting them earlier and easier, that usually indicates that, when two good teams face each other, they're fighting for the hardest and earliest clues. Thus, I think that if there are a large number of powers on a set, that doesn't do as well at differentiation teams as a harder set which operates with more room to differentiate them.
See, the problem is that your final sentence doesn't follow from any of your other premises. The fact that two good teams put up a lot of points on some packet does not indicate that the packet did not fairly distinguish between those teams.
It looks like from Tanay and Matt's posts that number of powers shouldn't be used to indicate the easiness of a set. I can accept that. So I guess I'll revise my opinion to: easy tournaments are less optimal at differentiation between top teams as harder tournament, bbut easy tournaments can still do it.
Even your weaker conclusion doesn't necessarily hold. Easy tournaments may or may not be optimal for differentiating between top teams, but that's an empirical rather than a theoretical question. If you want to do the research to show that this is the case, go ahead and compile that evidence. And even if we grant their suboptimality in this respect, it should be noted that nothing follows from this: if State College and LASA (two teams I am told are good at quizbowl?) throw down on an IS set at some point and the game is decided by 5 points, no one cares. When those teams go to national tournaments they'll get their chance to prove they are the best on harder questions.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
cvdwightw
Auron
Posts: 3291
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: "Regular" High School Difficulty

Post by cvdwightw »

Hey high schoolers (aka Kay Li) who are losing this thread more and more with every post: go read this post. Then go read this post. Then go read this thread. Then realize that not only has this whole discussion about difficulty been rehashed and resolved many times at the college level ("keep lead-ins and third parts hard enough to challenge the top teams, but make everything else easier to increase the scoring and fun at lower levels"), but that you are arguing in support of Ryan Westbrook.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry
User avatar
kayli
Auron
Posts: 1525
Joined: Sat Jun 21, 2008 10:15 pm
Location: Illinois

Re: "Regular" High School Difficulty

Post by kayli »

grapesmoker wrote:
Ar$oni$t$ Get All the Girl$ wrote:The gist of my argument is as follows: Teams are getting ridiculous amounts of powers. Powers are on the hardest parts of tossups and come earliest. More powers by teams indicates that they can get the hardest parts easier and earlier. If a set has too many powers, that means that the best teams are getting them earlier and easier. If teams are getting them earlier and easier, that usually indicates that, when two good teams face each other, they're fighting for the hardest and earliest clues. Thus, I think that if there are a large number of powers on a set, that doesn't do as well at differentiation teams as a harder set which operates with more room to differentiate them.
See, the problem is that your final sentence doesn't follow from any of your other premises. The fact that two good teams put up a lot of points on some packet does not indicate that the packet did not fairly distinguish between those teams.
This is fair. I guess you're right in that.
It looks like from Tanay and Matt's posts that number of powers shouldn't be used to indicate the easiness of a set. I can accept that. So I guess I'll revise my opinion to: easy tournaments are less optimal at differentiation between top teams as harder tournament, bbut easy tournaments can still do it.
Even your weaker conclusion doesn't necessarily hold. Easy tournaments may or may not be optimal for differentiating between top teams, but that's an empirical rather than a theoretical question. If you want to do the research to show that this is the case, go ahead and compile that evidence. And even if we grant their suboptimality in this respect, it should be noted that nothing follows from this: if State College and LASA (two teams I am told are good at quizbowl?) throw down on an IS set at some point and the game is decided by 5 points, no one cares. When those teams go to national tournaments they'll get their chance to prove they are the best on harder questions.
Unfortunately, it's hard to get good data from easier tournaments such as FNT because the best teams don't compete with one another at these kinds of tournaments. But I think I agree with most of what's said here. I think your last sentence was basically what I was trying to get at: that it's the harder tournaments that mean more because they're better at differentiating.
Kay, Chicago.
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: "Regular" High School Difficulty

Post by AKKOLADE »

It'd be fantastic if we could actually discuss this very important topic without people just posting the first thought that pops into their pretty little heads.

Thanks for making me waste 40 minutes the night before my lab practicals, chumps!
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
AlphaQuizBowler
Tidus
Posts: 695
Joined: Mon Dec 03, 2007 6:31 pm
Location: Alpharetta, GA

Re: "Regular" High School Difficulty

Post by AlphaQuizBowler »

This thread has gone places since my first post, so I guess I'd like to clear up what I was and wasn't saying.
grapesmoker wrote: ...I also appreciate the fact that people who work at the high school level on a regular basis (that would be folks like Fred, Charlie, and Matt Weiner) have a global view that you, Kay Li and William from Alpharetta, do not possess.
Which is why I stated this very clearly in my first post:
AlphaQuizBowler wrote:...I don't claim to have enough knowledge or experience to define high school difficulty. I just hope this thread can lead to a better dialogue about what different high school teams want (and need) out of high school questions.
I gave my opinion on the topic, and then attempted to start a substantial discussion about high school difficulty. This thread has not exactly turned out as I expected, so I definitely agree with Fred's sentiment:
Fred wrote:It'd be fantastic if we could actually discuss this very important topic without people just posting the first thought that pops into their pretty little heads.
Also, I never said the following things in this thread:
grapesmoker wrote:high school quizbowl is too easy
grapesmoker wrote:how awesome and how everything is totally easy


Anybody who thinks that IS-set/HSAPQ and GSAC are of the same difficulty level has not read the sets. I hope that we can at least agree that they're different. From there it's just a semantic issue: in my initial post, I preferred to label LIST/IS-set/HSAPQ "easy" and GSAC/BATE "regular." But I'm perfectly fine with labeling LIST/IS-set/HSAPQ "regular" and GSAC/BATE "harder than regular," if that's the convention that the larger quizbowl community prefers. At no point did I say that HSAPQ or NAQT should make their regular sets more difficult, in fact, I said the "easiness" of such sets was not a "problem, as I think the writers make a concerted (and largely successful) effort to make those tournaments extremely accessible." I did say "there is room in the high school season for 1 or 2 nationals warmup events on more challenging questions," but now I think that Fred makes a really good point about how question-writing resources are already stretched thin as it is.

Basically, my first post in this thread was about how we describe the difficulty of sets rather than any sort of judgment about the relative or absolute difficulty of high school quizbowl, such that I honestly don't know where statements like this one:
Fred wrote:[Y]ou must realize that making "regular difficulty tournaments" harder is the last thing you want to do; in fact, regular difficulty tournaments must become easier.

even came or who they were arguing with (though I again acknowledge that Fred makes a really good point about scarce resources in that same post).

Also,
Ar$oni$t$ Get All the Girl$ wrote:Chatahoochee has beaten Alpharetta in pretty much every meeting from what a quick search gives me.

You mean in the one game this whole season where our full A-teams have faced each other on real questions? Do a better search next time.
William
Alpharetta High School '11
Harvard '15
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: "Regular" High School Difficulty

Post by The Ununtiable Twine »

AlphaQuizBowler wrote:
Also,
Ar$oni$t$ Get All the Girl$ wrote:Chatahoochee has beaten Alpharetta in pretty much every meeting from what a quick search gives me.
You mean in the one game this whole season where our full A-teams have faced each other on real questions? Do a better search next time.
It appears that he did not know what your strongest lineup was. This coming from Mr. Li himself. "My bad," he adds.
Jake Sundberg
Louisiana, Alabama
retired
Locked