What We Talk About When We Talk About Difficulty

[Grams's Go-Go Boots]

Assorted thoughts regarding the difficulty of quiz bowl question sets in the form of a numbered list. I'd like to hear feedback on them. This is largely off the top of my head

1. The perceived difficulty for sets depends, in part, upon the quality and the experience of the team playing the set.

2. The actual effect of difficulty of a set on the statistics of teams is different for teams of differing strengths. Harder sets deflate the stats of the best teams less than they do less talented teams.

3. It is not impossible for difficult sets to be enjoyed by teams that are not very good.
3a. It is my opinion that there is a strong correlation between weaker teams that enjoy these harder sets and their experience with pyramidal quiz bowl; that is, the more often a team has played in the past, the more likely it is to be okay with playing a set that is harder than normal.

4. Some small percentage of trash in a quiz bowl set is fairly acceptable, depending on the tastes of the teams in the tournament's field. There's no reason to have a set standard for how much trash should be in a tournament when the fields of two tournaments may be completely different.
4a. Using trash to inflate statistics for a set, or using trash to "make up" for questions on overly difficult subjects, is not a good way to address the difficulty of a set. This is an academic activity, and at least the majority of it should be academic in nature. At the same time, it should not be forgotten that this is a leisure activity and a game, so it should be designed to be as enjoyable by as many players as is possible while still staying fair, rewarding knowledge, allowing the best team to win, etc.

5. Fairly related to 4: a hypothetical tossup with five lines of impossible clues that are followed by a giveaway that promises 100% conversion is not a good way to address the difficulty of a set. By not allowing teams to buzz in early on tossups, you're defeating the purpose of allowing teams to buzz in during the question.

6. There is a place in the market for some number of difficult sets. It's okay for these sets to be produced, though we should probably consider as a group just how many sets like this do need to be produced.
6a. The number of high school tournaments needing to be "harder than regular season difficulty" is probably far smaller than what was produced for the 2010-2011 season, and probably will be smaller than what is produced for the 2011-2012 season.
6b. This is probably going to be true for a long time, since teams that write their own sets are usually of pretty good quality and are interested in writing a set because they want to learn new things and improving as teams. When evaluating what they want to accomplish when writing a set, these teams are usually going to be much more willing to emphasize using harder clues (allowing them to learn new things) than easier ones, because that is a more direct benefit to the team writing the questions.

7. Overly difficult questions are not the leading reason that quiz bowl doesn't grow more, but it could be a limiting factor. Difficult questions are discouraging to some percentage of potential quiz bowl players. I personally believe that percentage would be fairly significant, though I have no data to back this up beyond personal experiences in discussing things with teams.

8. Of the "big three" subjects, on average, history and science tend to be easier than literature. This is because the curriculum for history and science classes tend to include more material commonly asked in quiz bowl sets than those for English/literature classes. This is, in part, because there's only a limited number of works that can be read in each literature class, while most history/science classes don't spend weeks at a time discussing one particular event in history/one particular law of chemistry.

[Mewto55555]

I think also possibly worth discussing in the same vein is whether or not "9. Difficult questions are needed to adequately distinguish between top teams." is true.

[abnormal abdomen]

I think also possibly worth discussing in the same vein is whether or not "9. Difficult questions are needed to adequately distinguish between top teams." is true.

I'm not sure who it was, but I once read a post by someone (I believe it was someone generally held in high regard within the quizbowl community) which basically indicated that, as long as the two top teams involved in a match aren't both buzzing on every first clue within a tossup, then they can't really complain about a set not differentiating between the two of them. The argument there would be that "if you knew clue Y about X thing then you would be able to buzz before the opposing team." I understand this, but I would think that this undermines, say, PACE, HSNCT, or ACF Nationals. I'm curious as to what the general consensus on this is. I may have just misread or misinterpreted what that poster was saying, though.

[Matt Weiner]

The whole point of pyramidal/good quizbowl is that we can ask questions on easy things with leadins that are challenging enough to distinguish top teams. If you are unable or unwilling to understand this then you do not understand good quizbowl and would have a hard time articulating why we bother with it as opposed to just playing Chip.

[Charbroil]

The whole point of pyramidal/good quizbowl is that we can ask questions on easy things with leadins that are challenging enough to distinguish top teams. If you are unable or unwilling to understand this then you do not understand good quizbowl and would have a hard time articulating why we bother with it as opposed to just playing Chip.

Whereas I agree with this, like Abid, I am curious in that case why it's necessary to make Nationals tournaments more difficult if you don't need harder questions to differentiate better teams.

[The Predictable Consequences]

The whole point of pyramidal/good quizbowl is that we can ask questions on easy things with leadins that are challenging enough to distinguish top teams. If you are unable or unwilling to understand this then you do not understand good quizbowl and would have a hard time articulating why we bother with it as opposed to just playing Chip.

Whereas I agree with this, like Abid, I am curious in that case why it's necessary to make Nationals tournaments more difficult if you don't need harder questions to differentiate better teams.

I don't think difficulty is at issue--you could make an extremely difficult tournament that doesn't differentiate among top teams because everyone's buzzing on the giveaway. I think the issue is more in terms of length, and the trade-off between difficulty differentiation and playability.

[in on these shenanigans]

I think also possibly worth discussing in the same vein is whether or not "9. Difficult questions are needed to adequately distinguish between top teams." is true.

If you want to make a question more difficult, there are two (and only two) ways to do it --

1. make early tossup clues (or hard bonus parts) harder
and/or
2. make the answer line a harder - that is, a less convertible - answer.

If you want to have more difficult questions to challenge upper-tier teams, you can do one or both, but keep in mind that the latter method ABSOLUTELY punishes weaker teams, while the other method is literally designed to educate while still giving the opportunity to buzz.

Therefore, it's not even a conclusion, but an axiom of good, pyramidal quizbowl that...

9. Distinguishing between top teams is done within the leadins of tossups and hard parts of bonus; distinguishing between middle teams is done within middle clues of tossups and medium parts of bonuses; distinguishing between weak teams is done within giveaways of tossups and easy parts of bonuses. The definitions of "top," "middle," and "weak teams" must be considered/determined by a tournament writer in considering the full audience that will hear these questions in competition. Questions on topics too difficult for this trichotomy to be employed are not appropriate for that level of play.

[Vernon Lee Bad Marriage, Jr.]

The whole point of pyramidal/good quizbowl is that we can ask questions on easy things with leadins that are challenging enough to distinguish top teams. If you are unable or unwilling to understand this then you do not understand good quizbowl and would have a hard time articulating why we bother with it as opposed to just playing Chip.

Whereas I agree with this, like Abid, I am curious in that case why it's necessary to make Nationals tournaments more difficult if you don't need harder questions to differentiate better teams.

So there are two fallacies in this kind of reasoning. First, the idea that harder questions, by mere virtue of being harder, are going to better distinguish between top teams, is flat-out wrong. A bonus that goes Blood Knot/Master Harold and the Boys/Athol Fugard is going to be way too hard for high school, but still isn't going to challenge any of the very best teams. Merely asking about the college canon is actually the worst of both worlds: it's boring to the best teams and totally inaccessible to the weaker ones. The ideal way to reconcile both is by asking about relatively underplayed things that you'd still have a good chance of knowing if you took an independent interest in the types of things quizbowl asks about. Generally, this means tossups on easy answers with new and interesting clues, as well as bonus parts on new aspects of things that get asked about a lot. I took this philosophy with the literature for the 2011 NSC. It had varying results, including some questions that in retrospect were too hard, but here's a bonus where I think it worked out pretty well:

18. This character claims that “Jesus shown everything off balance,” and he is accompanied by Hiram and Bobby Lee. For 10 points each:
[10] Name this character with a “scholarly look” who says another character would have been a “good woman if it had been somebody there to shoot her every minute of her life.”
ANSWER: The Misfit
[10] The Misfit murders Bailey, the grandmother, and their family on their trip to Florida in this Southern Gothic short story.
ANSWER: “A Good Man is Hard to Find”
[10] “A Good Man is Hard to Find” was written by this woman, who also wrote Wise Blood and “Everything that Rises must Converge.”
ANSWER: Flannery O’Connor
<Guth>

The first part isn't especially hard, in that it's something I could see people answering just because they'd read a famous story. However, I would contend that it does a better job of challenging good teams than an arguably harder bonus part on The Blood Knot. It uses clues that I would expect someone to remember if they had read that story, but not necessarily know just because they've been playing tons of quizbowl. By asking these kinds of questions, we actually challenge top teams better while simultaneously keeping things accessible for newer teams.

The second fallacy is the conventional wisdom that tournaments run on regular sets are "illegitimate." By design, national sets will produce a more precise gradation between the very best teams than a regular set might. Opening up the answer space also provides a greater and more interesting challenge for those teams. Nobody has denied either of those things. However, people seem to be ignoring that the national tournaments have vastly stronger fields than any other high school tournament during the year. At any other tournament, you'll be lucky to have one, maybe two, of the top 10 teams at PACE. In that vast majority of cases, you don't need that hyperfine gradation: a regular high school set will work just fine. Take the tournaments I played my senior year. I played State College and Maggie Walker many, many times on regular high school sets, and they kept winning. Even though all three of us were very good teams, these "illegitimate" questions somehow managed to make a clear distinction. The games weren't as interesting as they might have been at NSC, but both of those teams clearly showed they were a notch above us, even on the easier questions. An accessible set may produce a couple more buzzer races, but it will produce a perfectly legitimate result.

[abnormal abdomen]

The second fallacy is the conventional wisdom that tournaments run on regular sets are "illegitimate." By design, national sets will produce a more precise gradation between the very best teams than a regular set might. Opening up the answer space also provides a greater and more interesting challenge for those teams. Nobody has denied either of those things. However, people seem to be ignoring that the national tournaments have vastly stronger fields than any other high school tournament during the year. At any other tournament, you'll be lucky to have one, maybe two, of the top 10 teams at PACE. In that vast majority of cases, you don't need that hyperfine gradation: a regular high school set will work just fine. Take the tournaments I played my senior year. I played State College and Maggie Walker many, many times on regular high school sets, and they kept winning. Even though all three of us were very good teams, these "illegitimate" questions somehow managed to make a clear distinction. The games weren't as interesting as they might have been at NSC, but both of those teams clearly showed they were a notch above us, even on the easier questions. An accessible set may produce a couple more buzzer races, but it will produce a perfectly legitimate result.

Thanks, this is the sort of clear explanation I was looking for. To be clear, I wasn't arguing against pyramidal questions or anything earlier. I was basically just trying to get a better explicit understanding of why and how a "harder" set at nationals serves the aims of the tournament.

[Mooman]

The whole point of pyramidal/good quizbowl is that we can ask questions on easy things with leadins that are challenging enough to distinguish top teams. If you are unable or unwilling to understand this then you do not understand good quizbowl and would have a hard time articulating why we bother with it as opposed to just playing Chip.

Whereas I agree with this, like Abid, I am curious in that case why it's necessary to make Nationals tournaments more difficult if you don't need harder questions to differentiate better teams.

So there are two fallacies in this kind of reasoning. First, the idea that harder questions, by mere virtue of being harder, are going to better distinguish between top teams, is flat-out wrong. A bonus that goes Blood Knot/Master Harold and the Boys/Athol Fugard is going to be way too hard for high school, but still isn't going to challenge any of the very best teams. Merely asking about the college canon is actually the worst of both worlds: it's boring to the best teams and totally inaccessible to the weaker ones. The ideal way to reconcile both is by asking about relatively underplayed things that you'd still have a good chance of knowing if you took an independent interest in the types of things quizbowl asks about. Generally, this means tossups on easy answers with new and interesting clues, as well as bonus parts on new aspects of things that get asked about a lot. I took this philosophy with the literature for the 2011 NSC. It had varying results, including some questions that in retrospect were too hard, but here's a bonus where I think it worked out pretty well:

18. This character claims that “Jesus shown everything off balance,” and he is accompanied by Hiram and Bobby Lee. For 10 points each:
[10] Name this character with a “scholarly look” who says another character would have been a “good woman if it had been somebody there to shoot her every minute of her life.”
ANSWER: The Misfit
[10] The Misfit murders Bailey, the grandmother, and their family on their trip to Florida in this Southern Gothic short story.
ANSWER: “A Good Man is Hard to Find”
[10] “A Good Man is Hard to Find” was written by this woman, who also wrote Wise Blood and “Everything that Rises must Converge.”
ANSWER: Flannery O’Connor
<Guth>

The first part isn't especially hard, in that it's something I could see people answering just because they'd read a famous story. However, I would contend that it does a better job of challenging good teams than an arguably harder bonus part on The Blood Knot. It uses clues that I would expect someone to remember if they had read that story, but not necessarily know just because they've been playing tons of quizbowl. By asking these kinds of questions, we actually challenge top teams better while simultaneously keeping things accessible for newer teams.

The second fallacy is the conventional wisdom that tournaments run on regular sets are "illegitimate." By design, national sets will produce a more precise gradation between the very best teams than a regular set might. Opening up the answer space also provides a greater and more interesting challenge for those teams. Nobody has denied either of those things. However, people seem to be ignoring that the national tournaments have vastly stronger fields than any other high school tournament during the year. At any other tournament, you'll be lucky to have one, maybe two, of the top 10 teams at PACE. In that vast majority of cases, you don't need that hyperfine gradation: a regular high school set will work just fine. Take the tournaments I played my senior year. I played State College and Maggie Walker many, many times on regular high school sets, and they kept winning. Even though all three of us were very good teams, these "illegitimate" questions somehow managed to make a clear distinction. The games weren't as interesting as they might have been at NSC, but both of those teams clearly showed they were a notch above us, even on the easier questions. An accessible set may produce a couple more buzzer races, but it will produce a perfectly legitimate result.

I think the evidence goes both ways in this case. At QUAC with a HSAPQ set your senior year before your apotheosis, I remember us winning against you guys by around 20 points even though as you said we were probably a notch above you guys that year, and at ACF Fall Maggie Walker won against you by around 50, so I don't think the distinction was quite that clear as it was at later tournaments like PACE (despite my impressive four or five negs in that round against you at PACE).
I'm not saying HSAPQ's or ACF Fall's questions were bad (they certainly were not), but the difference in gradation I feel is somewhat larger than suggested, and I think you guys probably had a rather larger chance of winning the match at QUAC than at PACE.
Seeing that most questions probably went in the first few lines in those matches, it seems to me like the gradation playing on those questions would be similar to playing a national tournament with two to three line tossups. Sure, the better team will win more often on any set, but I feel that the difference between nationals sets and normal sets is rather pronounced.

[Kenneth Widmerpool]

To sort of explain what I was getting at in the HFT thread, I'll clarify my personal position:

1) I don't think that HFT needs to be as difficulty as any HSAPQ set. I think that if the set actually lived up to its stated difficulty (on the harder end of regular high school difficulty), that would have been fine. There would have been better differentiation between the top teams without resulting in 100-95 type games across the board for standard high school teams.

2) What I've noticed in HFTs past is that the difficulty increase mainly comes from asking questions that routinely come up in college sets. While reading HFT '10 to teams that have no business reading college packets at practice, I routinely cringed while reading tossups like that one on Heidegger that goes through his minor works before offering one measly clue about Being and Time and none about him being a Nazi collaborator. Like Matt said, this completely alienates most teams AND even runs the risk of boring the good ones. By using as its source material what comes up in college packets, HFT contributes to the "difficulty arms race" that a lot of high school teams get embroiled in. This becomes pretty tedious for all involved, not to mention increasingly exclusive.

3) I think that nationals sets do a good job at raising the difficulty level by exploring in-depth aspects of components of what high schoolers might be expected to know. This is where questions on social history, or entire bonuses on "A Good Man is Hard to Find," or characters from Shakespeare plays (just a few examples that I can think of) are things that teams are expected to have knowledge of, instead of the next named organic chemistry reaction or the fifth Cuban author.

I also don't think that the best high school teams currently hold a monopoly on all the knowledge of high school topics that even a regular difficulty tournament might be testing. I think that at this level, the best team wins: even if the team that loses has more absolute knowledge of all the clues in that particular round, the best team is the one that plays well, takes risks, knows when not to buzz, when to buzz, when to predict where a clue is going, and so forth.

[Cheynem]

Hmm, St Anselm's was a superplayoff team at PACE that year--I'm not particularly blanching if they come close to beating State College. I'll admit that regular difficulty tournaments sometimes have a difficult time gradating between the best of the best, but like, was every tossup a buzzer race after a line or something? Was every bonus an automatic thirty? How can you tell if it was the questions' fault that skills weren't being gradated or if it was just a round where St Anselm's (a great team) was playing particularly well?