Realistic assesment of difficulty

[Matt Weiner]

I think it's time that we started seriously considering how the difficulty of tournaments is advertised. I can think of few things more likely to scare away a team trying real quizbowl for the first time than a tournament that is way more difficult than it purports to be. When EFT is advertised as a "novice" tournament and delivers something indistinguishable from myriad regular-audience tournaments of the past several years, that gives people a very bad idea of what the community considers "novice" and discourages them from trying to play collegiate questions again in the future. When THUNDER is labeled "Regionals difficulty" and turns out to be basically indistinguishable from any ACF Nationals from 2000 to 2004 difficulty-wise, that's bad for the same reason. It's pretty obvious that most people, even a lot of people among those who understand every other aspect of question writing quite well, have absolutely no idea how to judge difficulty, either in an absolute sense, or, in what should be quite simple, a manner that is relative to past tournaments in an honest fashion. Until you figure out how to fix your thinking, I would suggest that you simply advertise your tournament as one level higher than whatever you think it is, so that you have a better chance of being correct.

[Mettius Fufetius]

When EFT is advertised as a "novice" tournament and delivers something indistinguishable from myriad regular-audience tournaments of the past several years, that gives people a very bad idea of what the community considers "novice" and discourages them from trying to play collegiate questions again in the future.

Well, I keep track of stats when I read packets, and I can tell you that I averaged roughly the same points per bonus practicing on 2007 EFT (~17) as my team achieved actually playing 2009 EFT. (For comparison, it was about 14 for 2006 MLK, which is what I would call a "regular-audience [tournament] of the past several years," and about 10 for this year's MO.) Comparing tossups is harder, but it's my subjective impression that there have always been a few rogue Regionals-level answer choices scattered around EFT. Frankly I think that's fine, as it gives new people a better idea of the canon than the more strictly circumscribed ACF Fall without actually being that much harder.

edit: words

[Mechanical Beasts]

On what data are you basing your opinions of THUNDER? While I have my gripes with the way some of the questions in all areas were written, that's obviously not it: and I can certainly say that the science was written to the difficulty level of '09 Regionals or slightly below that level. If that's also the difficulty of '00-'04 ACF Nationals, well, so be it, but that doesn't mean it's above Regionals difficulty in any sense.

Maybe the history and literature were a lot harder; I can say that the lit bonuses definitely seemed kind of hard. Then again, I think I had as many powers on lit as on science.

[Cheynem]

I found THUNDER to be highly fluctuating in difficulty. There were ACF Fall level tossups and then really hard stuff (especially in the Social Science and Religion components). Some of the Fine Arts was also, if not hard, rather...eccentric.

[MiltonPlayer47]

I thought this year's EFT was novice level in terms of the answer selection. The difference between this year's EFT and the past few ACF Fall tournaments was that the toss ups at EFT were written in a way to require knowledge more than list learning. In terms of bonus difficulty I thought it did the best job of any novice level tournament I've played. From what I remember it was pretty rare that there was a hard part of a bonus that was unreasonable for the advertised difficulty level. At least, that was my general impression of it.

[grapesmoker]

Thunder had some issues but to suggest that it's "basically indistinguishable from any ACF Nationals from 2000 to 2004 difficulty-wise," is just absurd. For the most part, the answer selection at this tournament was ok; there were a few wacky outliers and some bonus parts that could have used a reduction in difficulty, but the mass of the questions was not at all out of line with what you might expect to see at any recent ACF Regionals.

[Theory Of The Leisure Flask]

The music tossups at THUNDER were indeed very difficult; some of them would be among the harder questions at a 2009-vintage nationals. I felt that this was not true for the music bonuses, which if anything tended to be easier than other subjects on average, mostly due to the general lack of a "fuck you" third part in that subject (though there was plenty of variability).

More generally, though, I think I agree 100 percent with Matt regarding the general higher-than-advertised difficulty of tournaments these days. It's not always easy to realize when you've been playing for awhile and your personal knowledge has grown with the canon, but looking at all the THUNDER teams which sub-10 bonus conversions (which is a *lot* of them), it's clear that "regular difficulty" has evolved into something which could stand to be a bit more welcoming to novice players, who have distressingly few opportunities to play on good controlled-difficulty questions these days. This issue isn't unique to THUNDER or EFT though, rather it's endemic to all of circuit quizbowl.

[ClemsonQB]

The music tossups at THUNDER were indeed very difficult; some of them would be among the harder questions at a 2009-vintage nationals. I felt that this was not true for the music bonuses, which if anything tended to be easier than other subjects on average, mostly due to the general lack of a "fuck you" third part in that subject (though there was plenty of variability).

John Lawrence of Yale wrote the music for THUNDER, and while they may have been a tad too difficult (although looking over the answers it seems like about 2/3 of them wouldn't be out of place at a high school tournament, and the other third really aren't that difficult, certainly not what I'd label as at home among the harder questions at ACF Nationals), the content seemed good, especially for someone with little writing experience.

[setht]

I think it's time that we started seriously considering how the difficulty of tournaments is advertised. I can think of few things more likely to scare away a team trying real quizbowl for the first time than a tournament that is way more difficult than it purports to be. When EFT is advertised as a "novice" tournament and delivers something indistinguishable from myriad regular-audience tournaments of the past several years, that gives people a very bad idea of what the community considers "novice" and discourages them from trying to play collegiate questions again in the future. When THUNDER is labeled "Regionals difficulty" and turns out to be basically indistinguishable from any ACF Nationals from 2000 to 2004 difficulty-wise, that's bad for the same reason. It's pretty obvious that most people, even a lot of people among those who understand every other aspect of question writing quite well, have absolutely no idea how to judge difficulty, either in an absolute sense, or, in what should be quite simple, a manner that is relative to past tournaments in an honest fashion. Until you figure out how to fix your thinking, I would suggest that you simply advertise your tournament as one level higher than whatever you think it is, so that you have a better chance of being correct.

Leaving aside EFT for the moment because I didn't play/staff it and I don't feel like looking up stats right now, I think THUNDER was in fact a "Regionals difficulty" tournament consistent with Regionals and Penn Bowl of the last two years, with perhaps a few more fluctuations in difficulty--but as Jerry said, I think the bulk of the questions were in line with the advertised difficulty. In particular, comparing THUNDER stats with the Penn Bowl 2009 stats, I think bonus conversions for teams that had largely the same line-ups at both tournaments suggest that the THUNDER set as a whole was comparable to the Penn Bowl set. That certainly seems to be the case for the top teams, where there are a couple teams that had similar line-ups at both events. When I looked at the lowest bonus conversions from Penn Bowl 2009 at Penn and from THUNDER at Penn I couldn't find a team that had low bonus conversion at THUNDER and nearly the same line-up at Penn Bowl 2009. If people can point out some such teams I think that would be useful data for this discussion. In the meantime, I suspect that many of the low-level teams that show up to a low-level tournament in mid-October are rather different in strength from the low-level teams that show up to low-level tournaments in late January or February in general, and I suspect that is the case for Penn Bowl 2009 at Penn and THUNDER at Penn in particular. In other words, if the teams with the lowest BC at THUNDER had played on the Penn Bowl 2009 set, I'm not sure we'd be seeing better BCs from those teams.

looking at all the THUNDER teams which sub-10 bonus conversions (which is a *lot* of them), it's clear that "regular difficulty" has evolved into something which could stand to be a bit more welcoming to novice players, who have distressingly few opportunities to play on good controlled-difficulty questions these days. This issue isn't unique to THUNDER or EFT though, rather it's endemic to all of circuit quizbowl.

Is "regular difficulty" supposed to have a goal that true novice teams can hit 10+ ppb? I thought that particular goal was pretty much confined to novice events. If novice players aren't feeling welcomed when they attend regular difficulty tournaments, I think the circuit might want to focus more on (a) producing more novice tournaments, (b) literally welcoming novice players, emphasizing that tournaments have a social aspect beyond scoring points during matches, (c) mixing some older players in with the novices (for clubs that have the luxury of older players) to boost ppb so that the novices aren't scared off by getting sub-10 ppb.

-Seth

[Skepticism and Animal Feed]

THUNDER was a pretty easy tournament; I'm not sure it was even as hard as some recent ACF Regionals. There was some crazy stuff that came up, sure, but it was the exception and not the rule.

But I think that it would be a great tragedy if this thread devolved into a discussion of THUNDER's difficulty. For that is not the true point of Matt's initial post. Matt rather argues that people tend to under-estimate the difficulty of their tournaments, and his dropping of THUNDER is merely an example he wanted to use. Even though I disagree with Matt on THUNDER, I think his actual point might just be valid and should be discussed.

[Down and out in Quintana Roo]

This has been an increasing phenomenon at the high school level as well. It's one of the main reasons why i tend to distrust a lot of house-written sets.

[Theory Of The Leisure Flask]

THUNDER was a pretty easy tournament; I'm not sure it was even as hard as some recent ACF Regionals. There was some crazy stuff that came up, sure, but it was the exception and not the rule.

But I think that it would be a great tragedy if this thread devolved into a discussion of THUNDER's difficulty. For that is not the true point of Matt's initial post. Matt rather argues that people tend to under-estimate the difficulty of their tournaments, and his dropping of THUNDER is merely an example he wanted to use. Even though I disagree with Matt on THUNDER, I think his actual point might just be valid and should be discussed.

Yeah, my agreement with Matt was much more on the general point of tournaments being more difficult than advertised, especially for younger players- a situation that is especially unfortunate given the paucity of novice tournaments these days. THUNDER was no more difficult (and at times easier) than any other "regular difficulty" tournament, and the fact that roughly half the teams in attendance across all the mirrors failed to break 10 ppb was, unfortunately, not all that surprising. Numbers: of the sites which have stats up, 21 of 48 non-Florida teams were under 10; I excluded Florida because the high concentration of CC teams necesssarily means their field is going to be novice-heavy. This is, I think, strong empirical evidence that the tournament was "experienced player difficulty" rather than whatever "regular difficulty" should be, which is that level at which both novice and experienced teams can have meaningful, enjoyable games with clear differentiation of knowledge. When that concentration of players is pulling those conversion stats, perhaps it's time at the very least time to start producing more novice sets, because clearly causal and newer players are being underserved by the tournament mix- what's happening now, otoh, is that questions are being written by experienced players (mostly) for expereienced players, which is great or us experienced players but pretty shitty for someone who's entering college QB without a whole lot of college experience.

For what it's worth: I think the ideal "regular difficulty" that can produce rewarding quizbowl for the widest possible collegiate audience would lie halfway in between EFT and THUNDER. I'm not entirely sure it's possible to pull off, but I do think it's worth trying for at some point (I think this is what ACF Winter is supposedly shooting for; it's a shame that the timing of ACF Winter means that some teams will likely miss out due to still being on break).

[Skepticism and Animal Feed]

the paucity of novice tournaments these days

I dispute this.

We have:

ACF Fall (gets easier every year)
ACF Winter (never played it, but seems extraordinarily easy)
Minnesota Novice
Illinois Novice

Isn't there also another novice tournament that is being run in late November?

Also, this is if we buy into the "EFT is impossible" meme, which I personally don't buy.

[Matt Weiner]

ACF Winter was emphatically not "extraordinarily easy" last year. It was full of questions from the normal to above range, which were too long and difficult to read. It was damaging to bring people into that tournament on the premise that it would be "between Fall and Regionals" when it ended up being no such thing, and this was discussed at the time, as was the need to do better with the 2010 edition. This is exactly my point.

And I'll stand by what I said about THUNDER, though this thread is indeed not just about that tournament. Mixing in the occasional "what substance is made of hydrogen and oxygen and makes you wet" bonus part may skew the statistics to show an easier tournament, but when there's question after question that has no part that even a marginal Top 25 team knows, and scads of tossups on things that were middle or hard parts at ACF Nationals earlier in this decade, the tournament is hard. Sure, the top teams have learned a lot of things since then, so their numbers are fine, but I shouldn't have to explain yet again that the teams outside of the core 15-20 top ones do not study old packets and thus the idea of things becoming easier due to appearing in "the canon" does not apply to the vast majority of your audience for a tournament like this.

[Theory Of The Leisure Flask]

the paucity of novice tournaments these days

I dispute this.

We have:

ACF Fall (gets easier every year)
ACF Winter (never played it, but seems extraordinarily easy)
Minnesota Novice
Illinois Novice

Isn't there also another novice tournament that is being run in late November?

Also, this is if we buy into the "EFT is impossible" meme, which I personally don't buy.

Is a separate Illinois Novice actually happening? I wasn't aware it was on the calendar for this year (last year it was just a MUT mirror). I would call EFT newbie-friendly, certainly; maybe not a true "novice" tournament but obviously on the easy end. However, I wouldn't put ACF Winter in that basket- certainly going by what I've read of last year's questions, that seemed like pretty much a regular dificulty tournament (which after all is supposed to be pretty easy if you're a top player).

[Cheynem]

Illinois Novice, if I recall, was a MUT mirror last year.

[swwFCqb]

ACF Winter was emphatically not "extraordinarily easy" last year.

Matt is completely correct here. If you look at stats from last year's ACF Fall and ACF Winter (i.e. the mirrors my team attended), you will notice the PPB differential between the following teams:

Case Western: Fall-15.31; Winter-9.67 (one less player at Fall)
OSU A: Fall-17.89; Winter-11.05 (same exact team)
Ike: Fall-18.10; Winter-13.72 (solo for Fall, with Bob Kilner for Winter)

I'm sure the same pattern follows for other teams with similar personnel at both tournaments.

[ThisIsMyUsername]

The music tossups at THUNDER were indeed very difficult; some of them would be among the harder questions at a 2009-vintage nationals. I felt that this was not true for the music bonuses, which if anything tended to be easier than other subjects on average, mostly due to the general lack of a "fuck you" third part in that subject (though there was plenty of variability).

Sorry if my questions seemed off the mark (some of my teammates that attended THUNDER tell me there were some complaints about them at the Brandeis site, too). As George mentioned, it's my first time writing a category for a college tournament. I know we can't discuss the set on the boards yet, but if there were any particular questions you (or anyone else) felt were too difficult or problematic in any way, I'd be happy to hear feedback; you can e-mail me at john DOT lawrence AT yale DOT edu.

[grapesmoker]

Sorry if my questions seemed off the mark (some of my teammates that attended THUNDER tell me there were some complaints about them at the Brandeis site, too). As George mentioned, it's my first time writing a category for a college tournament. I know we can't discuss the set on the boards yet, but if there were any particular questions you (or anyone else) felt were too difficult or problematic in any way, I'd be happy to hear feedback; you can e-mail me at john DOT lawrence AT yale DOT edu.

I thought the "other arts" distro was completely misdistributed. I think you understand what I mean.

[ThisIsMyUsername]

I thought the "other arts" distro was completely misdistributed. I think you understand what I mean.

I'll take full responsibility for all answer choices and/or distributional problems for classical music, but I actually don't know what the misc. arts distribution ended up looking like. I was originally assigned just the 1/1 classical music per packet, and not the misc. arts. Later, I got an e-mail saying there were some sub-categories that they needed questions for and I wrote a couple of questions to fill those sub-categories.

[bmcke]

Maybe one issue is that a lot of elite players still play in stuff like EFT and Thunder. It warps the stats a little bit, because it looks like "plenty of teams got 20 ppb" when those teams weren't really part of the tournament's target audience.

[Matt Weiner]

Maybe one issue is that a lot of elite players still play in stuff like EFT and Thunder. It warps the stats a little bit, because it looks like "plenty of teams got 20 ppb" when those teams weren't really part of the tournament's target audience.

Well, for THUNDER, I think they are. It shouldn't be too much to ask that we have tournaments that can be played on by players of all levels; that's what tournaments normally should be, right? Segmenting everything into "novice" and "open/do whatever you want" is a markedly unsatisfactory approach.

For EFT, yeah, I think it would be nice if certain people didn't play it, and your overall point about how the stats are received is definitely a good one.

[grapesmoker]

I'll certainly say that my enjoyment of THUNDER would not have been affected at all if some of the difficulty issues with it had been corrected. I recognize that while I'm able (and happy) to answer questions on all sorts of wacky shit, most people are not.

[Magister Ludi]

I strongly agree with Matt about difficulty assessments in general and his thoughts about THUNDER. I had several negs in the tournament because I explicitly thought that a tossup answer was far too hard for "regular difficulty" and guessed something easier even though I knew the clue. I was so confused about this tournament's target difficulty at one point I comically negged a tossup by guessing Bulgakov's White Guard. This difficulty problem was especially prevalent in the music, which had all the strengths and weaknesses expected considering it was done by a first time writer with prodigious real knowledge.

[Mechanical Beasts]

I strongly agree with Matt about difficulty assessments in general and his thoughts about THUNDER. I had several negs in the tournament because I explicitly thought that a tossup answer was far too hard for "regular difficulty" and guessed something easier even though I knew the clue. I was so confused about this tournament's target difficulty at one point I comically negged a tossup by guessing Bulgakov's White Guard. This difficulty problem was especially prevalent in the music, which had all the strengths and weaknesses expected considering it was done by a first time writer with prodigious real knowledge.

I won't pretend that I didn't neg with A Pale View of Hills once. I certainly agree that there were several times where there were outlier tossups that affected how I and others played that set, but I don't think there were substantially more outliers than many other tournaments that purport to be regular difficulty (or some other difficulty). Even if there were, I think that it'd be more accurate to construe THUNDER as a less-well-controlled regular event rather than a very hard event.

[millionwaves]

Hey, everyone,

I'm following this thread with great interest, but I just wanted to clarify something:

I forgot to designate the questions that I wrote for the finals as such. So sometimes, things that were purposefully a little harder than the other questions, rather than being concentrated into a more uniformly difficult finals packet, were scattered throughout the rest of the questions haphazardly, which created some outliers. This is in no way the fault of the awesome people who helped us to randomize packets, and is entirely on me. Sorry to anyone whose enjoyment was thereby hindered!

[Cheynem]

This does explain a few things, but the difficulty uptick in some of these questions still strikes me as too extreme even for a finals packet.

[Susan]

I've split the discussion of novice tournaments to a separate thread.