Regular difficulty: does it exist?

This is the holding pen for the best threads containing quiz bowl talk.
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Regular difficulty: does it exist?

Post by Mechanical Beasts »

grapesmoker wrote:I don't understand what this argument is actually about. Look, we all speak the English language here, and we all know the meanings of the words "novice," "hard," and "regular." These labels are heuristics that help us determine what a given set is going to be like; of course, if we wanted to we could set some kind of conversion metric as the definition of these tournaments (as Matt likes to do). The benefit of the conversion metric is that we undershoot it, we know the set was too hard and if we overshoot it, it was too easy, when all tournaments using that set are considered. I guess this can be useful if you're trying to prove something quantitative about difficulty, but most of writing good tournaments for the appropriate audience is not about number crunching (although I suppose we might someday have the statistics that will tell us how conversion of tossups on Mind and Society has evolved over time), but rather about having experience, knowing the canon, and being able to predict what kind of questions people will be able to answer. That's a large part of what makes good writers good, their ability to consistently make that call correctly.
My objection from the beginning was: a new team's coming into the fold, and they need us to point them to a tournament that's "regular" for the circuit, and they don't really know what "regular" means (are most tournaments harder? easier?). Before I read the boards and was part of, like, quiz bowl at large, I simply didn't know that CO was hard for the circuit. I thought oh, this is a typical invitational; jeez, this is pretty hard, oh no!. We all seem content to point people to tournaments like ACF Regionals and say "this is what 'regular' means" and that's pretty much good enough for me.
I think I would be ready to concede Ryan's point that "regular difficulty" is not particularly well-defined. My rebuttal: so what? We all have some conception of what that means, and if we need to explain "regular difficulty" to people who don't know what it means, we just point to some tournaments that are considered emblematic of regular difficulty. I guarantee you that if we had to go through sets as a community and rank them on Jonathan Magin's scale, we would quickly come to some empirical consensus of what all these things mean. Trying to pin the tail on the difficulty donkey through abstruse debate is a pointless exercise.
Exactly! We have something to point to, so we're fine. I just wanted us to conclude something that could be pointed to, and justification for choosing it over other things. Sounds good.
Andrew Watkins
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Regular difficulty: does it exist?

Post by theMoMA »

Mrs. Rich wrote:
theMoMA wrote:But those tournaments are clearly in a different category from ACF Fall/EFT, and also clearly in a different category than Nationals/ICT/CO. I don't think anyone is trying to claim that all "regular difficulty" tournaments are the same, just that they're the questions that are between stuff explicitly written for novices and the stuff explicitly written to decide national championships, and are therefore the questions that regular tournaments are run on.
Saying that all "regular difficulty" tournaments do not have to be at roughly the same level of difficulty defeats the purpose of the label.
The purpose of the label is this: "Here is a tournament that falls within the range of what you can expect events written for normal competition to look like. This tournament is not written primarily for novices or for nationals-caliber competition." That's all that "regular" means, and that's plenty useful!
Andrew Hart
Minnesota alum
User avatar
DumbJaques
Forums Staff: Administrator
Posts: 3109
Joined: Wed Apr 21, 2004 6:21 pm
Location: Columbus, OH

Re: Regular difficulty: does it exist?

Post by DumbJaques »

As someone who is about to undertake the task of editing a "regular difficulty" tournament (perhaps, the original regular difficulty tournament), I kind of see both sides of the issue here. My personal conception of regular difficulty is pretty much synonymous with regionals, but I'm aware that certain regionals (07, for example) were definitely harder than regular difficulty; consequently, I'm unsure of how others will interrpret even something as seemingly rigidly defined as "Regionals difficulty," so in the past I've said "regular difficulty." Clearly that doesn't help at all. It's clear that ACF is now dedicated to (among many other positive things) avoiding inconsistencies in difficulty, so I look forward to just being able to say "Regionals" or "Winter" or "Fall" or "Nationals" - or whatever appropriate ranges in between - in all future tournament evaluations. I tend to think this is the best way to go anyway.

Something else I wonder about is the whole issue of changing standards. I don't mean it in this case so much in terms of quality, but rather what's accepted as easy, hard, canon-pushing, etc. The chain of quizbowl being that begins with clues/bonus third parts and ends as tossup answers goes further; things that were previously only workable as tossups at the masters level even a few years ago are now often more than acceptable at something like Regionals, or TIT, or Stanford, etc. So, there's another layer of problems to using terms like "regular difficulty," because that shifts every year.
Chris Ray
OSU
University of Chicago, 2016
University of Maryland, 2014
ACF, PACE
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Regular difficulty: does it exist?

Post by Matt Weiner »

DumbJaques wrote:Something else I wonder about is the whole issue of changing standards. I don't mean it in this case so much in terms of quality, but rather what's accepted as easy, hard, canon-pushing, etc. The chain of quizbowl being that begins with clues/bonus third parts and ends as tossup answers goes further; things that were previously only workable as tossups at the masters level even a few years ago are now often more than acceptable at something like Regionals, or TIT, or Stanford, etc. So, there's another layer of problems to using terms like "regular difficulty," because that shifts every year.
Not everyone believes in this model of so-called canon expansion, and it's an issue that is intertwined with the notion of who tournaments are for and how to properly measure difficulty. One cannot argue one's way out of the fact that a tossup on, e.g., "A Modern Instance" is too hard for a regular-difficulty tournament if only 20% of rooms convert the tossup. The fact that it's come up 100 times before and that any serious academic player will know at least 5 Howells works cold means precisely nothing when faced with the cold reality of what percentage of games actually answer the question. The "today's bonus parts are tomorrow's tossups" model has no relevance to anyone outside the small percentage of quizbowl players who consciously try to predict and prepare for what the next tournament's questions will be.
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
User avatar
No Rules Westbrook
Auron
Posts: 1238
Joined: Mon Nov 22, 2004 1:04 pm

Re: Regular difficulty: does it exist?

Post by No Rules Westbrook »

Not everyone believes in this model of so-called canon expansion, and it's an issue that is intertwined with the notion of who tournaments are for and how to properly measure difficulty. One cannot argue one's way out of the fact that a tossup on, e.g., "A Modern Instance" is too hard for a regular-difficulty tournament if only 20% of rooms convert the tossup. The fact that it's come up 100 times before and that any serious academic player will know at least 5 Howells works cold means precisely nothing when faced with the cold reality of what percentage of games actually answer the question. The "today's bonus parts are tomorrow's tossups" model has no relevance to anyone outside the small percentage of quizbowl players who consciously try to predict and prepare for what the next tournament's questions will be.
This is the type of mentality that I find irksome. For one thing, you say "players who consciously try to predict and prepare" - to me, that's the definition of what it means to be a valid collegiate qb player - everyone should be doing that. But, look, we all understand that there's a certain difficulty level that's too hard for any given tournament - a certain level beyond which the game doesn't play well or teams are driven away or other bad stuff happens - and we can argue and have different points of view about where that line is for any given event.

But, what I'm arguing for is some truth-in-advertising when it comes to saying what difficulty certain things are. You're never going to convince me that a tossup on bremmstrahlung is hard, no matter how few teams answer it at whatever event it appears at - all you're going to convince me of by pointing at low conversion numbers is that the teams at that tournament were not very good. I can call something "easy" or "average" and still know that lots of teams at a given event aren't going to know it - I don't buy into the Teitlerian argument that there is no reasonable standard of difficulty apart from conversion numbers. I think the reasonable standard, as I've said before in this thread, is the considered judgment of experienced players who know the canon and who know how hard certain things are to learn, etc.

I just think it's a lot more honest and less euphemistic and pandering to call things by this standard - saying something is gettable or average does nothing in my system to imply that people are going to answer it, or even to imply that it was appropriate for a given tourney all things considered. It's just calling a spade a spade.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!
User avatar
magin
Auron
Posts: 1005
Joined: Fri Oct 27, 2006 5:50 pm
Location: College Park, MD

Re: Regular difficulty: does it exist?

Post by magin »

Yeah, this whole debate over "regular difficulty" seems unnecessary. It seems to me that there are three basic levels of difficulty: novice (associated with tournaments like EFT and ACF Fall), regular (associated with Penn Bowl, MLK, Terrapin, and ACF Regionals), and hard (associated with ACF Nationals, NAQT ICT, and Chicago Open). However, all these measures of difficulty take place after the fact; if I advertise a tournament as having novice difficulty, and proceed to write lots of tossups on figures like Elias Canetti and Salvatore Quasimodo, the tournament is actually quite difficult. Using "regular difficulty" is just a convenient shorthand for tournaments more difficult than novice events but less difficult than hard events.

Also, Ryan, I'm confused about what you're trying to accomplish here. Do you want Matt and other editors to produce tournaments at the same difficulty level they would otherwise, and just not call them "regular difficulty"? Do you want those events to be more difficult? Just to let you know, the goal for the 2009 ACF Regionals is for it to be enjoyable for as many teams as possible while still enabling teams who know more to demonstrate that knowledge through early clues and second/third bonus parts, which means that it will resemble last year's Regionals more than the Regionals of 2006 and 2007.
Jonathan Magin
Montgomery Blair HS '04, University of Maryland '08
Editor: ACF

"noted difficulty controller"
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Regular difficulty: does it exist?

Post by Matt Weiner »

No Rules Westbrook wrote: I don't buy into the Teitlerian argument that there is no reasonable standard of difficulty apart from conversion numbers.
Well, then you're buying into some truly bizarre notion that there is a Platonic Form of difficulty out there in the ether, and abandoning any sort of good sense about looking at the world through objective empirical evidence...it's troubling as either an editorial philosophy or a metaphysical one.
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
User avatar
No Rules Westbrook
Auron
Posts: 1238
Joined: Mon Nov 22, 2004 1:04 pm

Re: Regular difficulty: does it exist?

Post by No Rules Westbrook »

I don't understand Magin (and Jerry and others too) and their argument that this debate is over nothing. Clearly, there are arguments to be made on both sides here.

Lots of people keep saying that it's apparent that there are levels of difficulty like novice, harder than novice but easier than nationals, and nationals-level. Well, yeah, that's apparent - but it's not apparent how hard any of those levels is - the definition of those things that you keep saying is not apparent at all, unless you just baldly accept someone else's definition. What standard of difficulty are you using? - the one of today or the one of 2006? Clearly, it makes a big difference - noone denies that nationals and regionals have expereinced quite a shift in difficulty (again, this argument is not about whether that shift is good or bad). But, you have to tell me why you're using the standard that you are - why the one of today instead of the one of yesterday? Just because it's the current one? Why not invent a new one and use that?

In response to Matt's Platonic Form criticism, yes - I do think that experienced players who have a solid grasp of the canon and a knowledge of how difficult things are to learn and acquire can make reasonable arguments about difficulty. Sure, we're not always going to agree, or even usually going to agree since we have different biases - but there are reasonable arguments to be made. I'm arguing that your system focusing exclusively on conversion is more relativist than mine, and hence more skeptical - so I could turn your metaphysics reference around on you and say that you're the one living in a constantly changing universe without solid foundational principles of how to evaluate difficulty.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: Regular difficulty: does it exist?

Post by grapesmoker »

No Rules Westbrook wrote:I don't understand Magin (and Jerry and others too) and their argument that this debate is over nothing. Clearly, there are arguments to be made on both sides here.
My point is that I don't understand what the "other side" (that's you) is supposed to be arguing. Actually, I don't understand completely what the argument is about at all, other than that you think that "regular difficulty" is a somehow deceptive term.
Lots of people keep saying that it's apparent that there are levels of difficulty like novice, harder than novice but easier than nationals, and nationals-level. Well, yeah, that's apparent - but it's not apparent how hard any of those levels is - the definition of those things that you keep saying is not apparent at all, unless you just baldly accept someone else's definition.
Bolding mine. I have no idea what it is that you're arguing here. We've delineated degrees of relative difficulty for you, and you keep asking "but how hard is it, really?" This is a strange question because there is no absolute difficulty scale for how "hard" things are; as a rule, things are grasped more easily proportional to how much they are repeated, and when you get down to it, the reason a question about Henry II of England is easier than the question about Sigismund II of Poland is that the former tends to be more heavily emphasized in the sorts of situations where one typically learns about those things, such as intro history classes.

By the way, I should like to state here that my position regarding difficulty is firmly linked to what's covered in college classes. Many people have argued that the QB canon shouldn't be referenced to college curricula (e.g. Matt Weiner); I hold that the QB canon is already referenced to the college curricula to various (large) degrees depending on the disciplines, and furthermore that this is actually a desirable thing. I suppose this puts me in some sort of third position at odds with both Ryan and Matt, and I'll be happy to articulate this theory further if anyone cares to hear it.
What standard of difficulty are you using? - the one of today or the one of 2006? Clearly, it makes a big difference - noone denies that nationals and regionals have expereinced quite a shift in difficulty (again, this argument is not about whether that shift is good or bad). But, you have to tell me why you're using the standard that you are - why the one of today instead of the one of yesterday? Just because it's the current one? Why not invent a new one and use that?
Again, there is some serious confusion here. First of all, the difference between ACF Regionals 2006 and 2008 is not nearly as large as some would make it out to be; in fact, Regionals has varied somewhat in difficulty throughout its existence, and as Zeke recently noted, many of those old questions were very good and clue-dense and would be challenging for today's players. Second, I have no idea how I might designate a particular question as falling into Regionals 06 but not Regionals 08 (or vice-versa) difficulty unless those questions actually appeared in those tournaments.
In response to Matt's Platonic Form criticism, yes - I do think that experienced players who have a solid grasp of the canon and a knowledge of how difficult things are to learn and acquire can make reasonable arguments about difficulty. Sure, we're not always going to agree, or even usually going to agree since we have different biases - but there are reasonable arguments to be made. I'm arguing that your system focusing exclusively on conversion is more relativist than mine, and hence more skeptical - so I could turn your metaphysics reference around on you and say that you're the one living in a constantly changing universe without solid foundational principles of how to evaluate difficulty.
Mind status: blown. Furthermore, I think many of those words do not mean the things that you think they mean.

Hey, as long as I'm at it, I'm going to point to one particular example that you, Ryan, used, and I'm going to break it down. I'm referring specifically to the statement you made to the effect that no one could convince you that "brehmstrahlung" is a difficult answer. Now, I feel like I'm pretty well-qualified to speak on this topic, since I'm probably one of fewer than 5 current QB players who has actually derived the brehmstrahlung formula. So: this is not a topic that most physics majors would be exposed to until they covered advanced classical electrodynamics. For most majors at Berkeley, which I'll take to be fairly representative, that would happen in the second semester of one's junior year. So how does it come to be that even though people who ostensibly study the subject would have a high probability of not knowing about brehmstrahlung radiation than noted non-physicist Ryan Westbrook? The answer, of course, is obvious: this happens because of the way that quizbowl is structured and the fact that physics questions need to be written for quizbowl tournaments, so sooner or later named concepts are bound to come up. Since you're a good quizbowl player, you tend to remember the clues for those tossups, and that's how you come to think of brehmstrahlung as an easy answer when in fact it's a relatively advanced topic for upper level physics students.

The point of this analysis is not to justify or castigate the writing of tossups on bremstrahlung at various levels; rather it is to demonstrate that to a large extent, what's considered easy or hard in quizbowl is largely dependent on what's come up before, and those things themselves are eventually pegged back to what people learn about while studying various topics. The larger point is that the entire canon is, in some sense, demarcated by empirical boundaries of gettability rather than by any abstract notion of what's hard and what's not; in fact, there is no absolute measure of such a thing, as I have been trying to demonstrate.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Regular difficulty: does it exist?

Post by Mechanical Beasts »

grapesmoker wrote:The point of this analysis is not to justify or castigate the writing of tossups on bremstrahlung at various levels; rather it is to demonstrate that to a large extent, what's considered easy or hard in quizbowl is largely dependent on what's come up before, and those things themselves are eventually pegged back to what people learn about while studying various topics. The larger point is that the entire canon is, in some sense, demarcated by empirical boundaries of gettability rather than by any abstract notion of what's hard and what's not; in fact, there is no absolute measure of such a thing, as I have been trying to demonstrate.
This reminds me of the good old days: some of the first bitching I ever did about a tossup, justified or otherwise! Some common-link tossup on reductions (I want to say this was ICT this year) led in with "something something the Meerwein-Pondorf-Verley one" and I buzz and get fifteen points. I complained, saying that that was "too easy" or something, and Eric said that I probably just remembered it as an ACF Regionals 2007 leadin. I had done no such thing; I probably encountered it from reading something unrelated to quizbowl. But this underscores your point: gettability isn't necessarily based on the past, though certainly for many players the past is a defining marker of gettability.
Andrew Watkins
User avatar
cvdwightw
Auron
Posts: 3291
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Regular difficulty: does it exist?

Post by cvdwightw »

I think there is a major flaw in Ryan's argument, and it concerns the idea that the canon is all there is to know. Arguing that "if it's in the canon, people should know it" implies that quizbowl is an Academic Decathlon-like competition with a rigidly defined (if more expansive) set of things that should be known and a set of things that are unimportant to know because they don't have to be known.

To me, the canon serves as a loose guide to what people do know, rather than a depository for what people should know. It appears to me that Ryan assumes the existence of some mythical group of players who learned everything they know by reading old packets, and this is absurd. As long as people continue to learn things outside of quizbowl, the canon can only serve as a rough approximation of what an average person at a certain skill level is likely to know.

Furthermore, because people tend to learn the majority of things via college curricula, I agree with Jerry that the canon must inherently be rooted in such a thing. Certainly the college curriculum is not the final arbiter of what is and what is not canonical, especially in such things as literature where the obscurity of a book is in no way related to where one might encounter it, but at the lowest level of the canon, the Novice/Fall difficulty level almost entirely consists of things people will be likely to have encountered in their high school or "Intro to..." classes.
No Rules Westbrook wrote:Lots of people keep saying that it's apparent that there are levels of difficulty like novice, harder than novice but easier than nationals, and nationals-level. Well, yeah, that's apparent - but it's not apparent how hard any of those levels is - the definition of those things that you keep saying is not apparent at all, unless you just baldly accept someone else's definition. What standard of difficulty are you using? - the one of today or the one of 2006?
As I have stated earlier in the thread, ACF Fall, Winter (when it happens), Regionals, and Nationals of the previous year are the standard of difficulty for the difficulty levels of Novice/Fall, Winter, Regionals, and Nationals. Each of these tournaments is edited by a number of experienced players and editors who estimate both what players should know (Ryan's argument) and what they actually do know (Matt's argument). To create a certain difficulty level, these editors combine their intuition about what should be known with empirical data about what actually was known the last time this event happened. If they do their job correctly, then there is little difference between what is known and what should be known at that level.

Between each of these four pillars of difficulty lie three intermediate difficulty levels, which I have termed "Easy", "Regular", and "Hard". These three levels, as well as "Post-Nationals", are entirely determined by the difficulty of ACF Fall/Winter/Regionals/Nationals.

So, essentially, I am trusting a cadre of experienced players and editors to determine what constitutes each of four levels of difficulty based on a combination of what should be known and what is known to be known. Every other mACF tournament is expected to then use these four tournaments as benchmarks for difficulty. When the next ACF Fall comes around, it uses the old ACF Fall data as an indicator of what is known to be known, but it also replaces the old ACF Fall as the new standard for Novice/Fall difficulty. Based on this, we can make objective statements about the difficulty of a tournament - if a bunch of teams that are getting around 10 ppb at ACF Winter and 15 at ACF Fall end up with 13 ppb at some random tournament, then the tournament should have been labeled "Easy". If it wasn't, then the tournament missed its stated difficulty level, and if and when those packets get send to the archives, they should be labeled as "Easy" difficulty level rather than the advertised difficulty level.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: Regular difficulty: does it exist?

Post by grapesmoker »

Incidentally, and on a somewhat unrelated note, I'd like to offer myself as an example of someone who does not know 5 William Dean Howells works down cold. It just never occurred to me to learn much about any of them. Which underscores another point that I wanted to make: would Howells be the unofficial official author of ACF if it hadn't been for Andrew and Zeke back in the day, who I believe made something of a special project of getting him more exposure? Just another example of how quizbowl evolves in rather erratic ways and what's considered "easy" is very dependent on the course of that evolution.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
setht
Auron
Posts: 1205
Joined: Mon Oct 18, 2004 2:41 pm
Location: Columbus, Ohio

Re: Regular difficulty: does it exist?

Post by setht »

I'd like to start off my post/attack on Ryan Westbrook by juxtaposing the following pair of statements:
No Rules Westbrook wrote:You're never going to convince me that a tossup on bremmstrahlung is hard, no matter how few teams answer it at whatever event it appears at
and
No Rules Westbrook wrote:I prefer some sort of fuzzy definition of difficulty that doesn't look at empirical stats (the Teitler approach, as I always call it), but uses the judgment of more experienced players on what's come up before and what is hard to know and what is easy to know and so on - I think that this approach is way more accurate in the long run.
Ryan, let me join Jerry in trying to convince you that a tossup on bremsstrahlung is hard without discussing how few teams answer it at any particular event: there certainly have been a fair number of bremsstrahlung tossups in the past, so it has "come up before", but under no reasonable definition of "what is easy to know" would bremsstrahlung qualify as "easy to know" (aside from the butt clue on the German translation that always appears as the giveaway, I guess). You have picked out an example of something that I am confident almost no one in college or in quizbowl really knows anything about--I'm very confident that the vast majority of quizbowlers can't name two facts about bremsstrahlung aside from the translation bit, possibly not even one fact. I certainly didn't learn anything about bremsstrahlung in the first several years of science classes I took, and I'm pretty sure that's true for all science students--it's just not something that shows up in lower-level classes.

I'm going to assume you're prepared to concede that a tossup on bremsstrahlung is in fact difficult. Now, let's suppose you never pay attention to how bremsstrahlung play in your rooms at tournaments or look at any other empirical evidence for how questions play out. After all, you're a very experienced player and you already know everything you need to know about the difficulty of tossups on bremsstrahlung, right? Wrong. As I said above, I think it's possible that a large number of quizbowlers can buzz on "from the German for 'braking radiation'", but I think (and I really hope you'll agree) that any tossup that sits in 90+% of all rooms and only gets picked up on the giveaway, very possibly in a buzzer-race in lots of rooms, is in fact a failed tossup.

What does this say about your judgment of difficulty? What does this say about your proposal to define difficulty using only the opinions of experienced players such as yourself without reference to empirical data?

Let's move on to your "Platonic forms of difficulty" bit. Now, if I understand you correctly, you're saying that there exists some perfect standard of, for example, "ACF Nationals" difficulty which has nothing to do with the field strength at any particular ACF Nationals event. To some extent, I sympathize with this notion when it's put that way--I think ACF Nationals is an event that should largely be written to challenge the top-level teams in the country. If ACF Nationals 2009 has 5 top-level teams and 50 low-level teams, I still want to see the playoffs and finals be written at a level that is challenging for the top-level teams. But notice that I'm not completely divorcing the target difficulty from the target audience: I'm just saying the target audience is a particular subset of the field. If ACF Nationals 2009 has 0 top-level teams and 50 low-level teams, I don't see any point in writing playoff and finals packets that just go over everyone's heads. I get the impression that if ACF Nationals 2009 does have 0 top-level teams you'd still like to see packets written at your Platonic ACF Nationals level of difficulty, and I don't understand why you'd want that.

Let me see if I can highlight the ludicrousness of this position by turning things around a bit: suppose now that ACF Nationals 2009 draws 50 teams for whom your current Platonic ACF Nationals level of difficulty is in fact very easy. If the tournament is written at your current ideal level, these teams will 30 every bonus without thinking hard, buzz on the first line of each tossup, etc. Supposing you know all this in advance--would you still insist that the tournament should be written at your current Platonic ACF Nationals level of difficulty? If you update your Platonic notion of ACF Nationals level difficulty to something that will challenge these teams in much the same way that ACF Nationals 2008 challenged the field at ACF Nationals 2008, have you not in fact explicitly based your notion of difficulty on the strength of the field? I acknowledge that this example is ridiculous, will never happen in practice, etc. The point I'm trying to make is that I think it makes sense to peg the target difficulty of a tournament set to the perceived strength of the target field. I further think that even experienced writers and editors like yourself can benefit from paying attention to conversion statistics and incorporating them into future attempts to predict field strengths and produce questions of an appropriate difficulty. What is there to object to in all this? Have you bought into your self-ranking as the #1 question writer of the game so much that you feel you no longer have room to improve?

-Seth
Seth Teitler
Formerly UC Berkeley and U. Chicago
President of NAQT
Emeritus member of ACF
User avatar
No Rules Westbrook
Auron
Posts: 1238
Joined: Mon Nov 22, 2004 1:04 pm

Re: Regular difficulty: does it exist?

Post by No Rules Westbrook »

Jerry, if you want to add "whether stuff comes up in college courses" to the general formula for deciding whether something is difficult, that's reasonable - I'm sure there are other factors you can add to that formula too. Like you say, that particular factor will apply more to certain answers than others, and sometimes it will need to be used with discretion, but it's a reasonable thing to use.

Dwight, I don't think I have a problem with the model you've elucidated. I just don't think it's what happened in reality at all. With regard to recent regionals and nationals, a group of people got together and decided "we're going to make this easier to accomadate more teams." That decision wasn't based on some combination of my method and Weiner's method - it was a pragmatic move intended to bring in more teams and increase accessibility (which I'm not saying is good or bad). In the same way, I think the current definition of "Easy" and "Hard" hasn't evolved by some natural progression like you describe - rather, it's been perverted and twisted on purpose, such that the way those terms are used now by some people makes them inaccurate.

Plus, if you haven't noticed, I'm pretty much the only one arguing for "Ryan's method" here, so I'd be about the only one to change it that way.

To Magin and anyone else who seems not to understand what I'm arguing, it's not rocket science. I'm saying - Stop calling stuff "Hard" if it isn't. Stop using words like "regular" when they're deceptive. Of course, that won't happen, but that's all I'm arguing. I'm sure plenty of people disagree (or think there's nothing to be gained in such theoretical distinctions), but I disagree. Oh, I see Seth has posted, I'll respond to him separately.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Regular difficulty: does it exist?

Post by Matt Weiner »

No Rules Westbrook wrote:With regard to recent regionals and nationals, a group of people got together and decided "we're going to make this easier to accomadate more teams."
....by bringing Regionals into line with Penn Bowl, MLK, Terrapin, and the other normal tournaments people play. The goal was to eliminate the notion of "Regionals difficulty" as some special thing that existed just for that one tournament, and kick it back to the difficulty of regular tournaments. The new difficulty for Regionals wasn't picked up arbitrarily, it was a conscious effort to emulate all those other tournaments.
Stop using words like "regular" when they're deceptive.
I just don't see how "regular," defined as "the bulk of tournaments that non-pussy quizbowl teams do in fact play," is in any way legitimately confusing or misleading.
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: Regular difficulty: does it exist?

Post by grapesmoker »

No Rules Westbrook wrote:Plus, if you haven't noticed, I'm pretty much the only one arguing for "Ryan's method" here, so I'd be about the only one to change it that way.
I enjoy random theoretical discursions about the game, and I think you do a better job as a non-ironic devil's advocate than ever could, so this discussion should be valuable to people as an expose of what random experienced editors actually think, if for no other reason.
To Magin and anyone else who seems not to understand what I'm arguing, it's not rocket science. I'm saying - Stop calling stuff "Hard" if it isn't. Stop using words like "regular" when they're deceptive.
There's a reason we don't understand what your argument is supposed to be about, and it's not because we're retarded. Stuff is hard if a lot of people don't know it, and if they do, then it isn't hard anymore; also, there's nothing deceptive about the notion of regular difficulty, since I am sure that if a whole bunch of editors got together and decided to rate the difficulty of many tournaments on a 10-point scale, we would soon arrive at a reasonable definition of regular difficulty.

edit: you'll all know the Kronig-Penney model once I'm done with you!
Last edited by grapesmoker on Tue Aug 26, 2008 1:35 am, edited 1 time in total.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
No Rules Westbrook
Auron
Posts: 1238
Joined: Mon Nov 22, 2004 1:04 pm

Re: Regular difficulty: does it exist?

Post by No Rules Westbrook »

I get the impression that if ACF Nationals 2009 does have 0 top-level teams you'd still like to see packets written at your Platonic ACF Nationals level of difficulty, and I don't understand why you'd want that.
Okay, let me start out by saying that I am emphatically not arguing this. I'm not talking about how packets should or should not be written, at all. Packets should be written for a target audience. I'm arguing that, in a Nationals with 0 top-level teams, we shouldn't call the easier packets "Hard." We should call them "Average" or some other word - that doesn't mean they're not appropriate for Nationals, maybe they're appropriate for that Nationals. Like I said before, and I've always said it, there's a line of difficulty that it makes no sense to go beyond. But, that doesn't mean the questions you've created are "Hard" just because it's the current level. I'm arguing for truth-in-naming - that's it.

I am also emphatically not arguing that, in editing a tournament, you shouldn't consider field strength. Of course you should. That should be the paramount consideration in editing a tournament - are these questions appropriate for the audience. But, I'm not going to go around pretending that those questions are "hard" or use some term like "regular difficulty" when I think those terms aren't appropriate.

I'll give an example. If a tossup on the Bear Flag Revolt goes dead (I'll talk about history since I have a greater feel for that than science), I'm not going to accept the proclamation "that tossup was too hard." Rather, I'm going to say "That's a tossup of reasonable difficulty - but the teams playing this event didn't know it, so it wasn't appropriate for this event." I'm arguing that this is a meaningful distinction.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!
User avatar
No Rules Westbrook
Auron
Posts: 1238
Joined: Mon Nov 22, 2004 1:04 pm

Re: Regular difficulty: does it exist?

Post by No Rules Westbrook »

Yeah, I'll concede Weiner's point that "regular" obviously acquires some concrete meaning after it's been around awhile. But, I think "regular difficulty" is a euphemism - that's the best term - and I don't like euphemisms.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!
User avatar
cvdwightw
Auron
Posts: 3291
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Regular difficulty: does it exist?

Post by cvdwightw »

No Rules Westbrook wrote:I'm arguing that, in a Nationals with 0 top-level teams, we shouldn't call the easier packets "Hard." We should call them "Average" or some other word - that doesn't mean they're not appropriate for Nationals, maybe they're appropriate for that Nationals. Like I said before, and I've always said it, there's a line of difficulty that it makes no sense to go beyond. But, that doesn't mean the questions you've created are "Hard" just because it's the current level. I'm arguing for truth-in-naming - that's it.

I am also emphatically not arguing that, in editing a tournament, you shouldn't consider field strength. Of course you should. That should be the paramount consideration in editing a tournament - are these questions appropriate for the audience. But, I'm not going to go around pretending that those questions are "hard" or use some term like "regular difficulty" when I think those terms aren't appropriate.

I'll give an example. If a tossup on the Bear Flag Revolt goes dead (I'll talk about history since I have a greater feel for that than science), I'm not going to accept the proclamation "that tossup was too hard." Rather, I'm going to say "That's a tossup of reasonable difficulty - but the teams playing this event didn't know it, so it wasn't appropriate for this event." I'm arguing that this is a meaningful distinction.
See, now I think we're getting somewhere. We're comparing two different things. Ryan is arguing for an objective difficulty. I'm not sure what Ryan's criteria are for placing questions into "easy", "medium", and "hard" categories, but presumably a "hard" tournament would contain mostly "hard" questions. Everyone else is arguing for a subjective difficulty - relative to field strength and mainstream tournaments, is this an easy tournament, regular difficulty, or hard tournament - whereas Ryan calls this "appropriate" and claims that it's independent of his objective scale of difficulty. If a question on the Bear Flag Revolt goes dead in 80% of rooms, the tossup may not be "hard" on Ryan's scale of difficulty, but it is definitely "too hard" for the field. While Ryan bases his difficulty on how many questions fit his definition of hard, everyone else bases their difficulty on how many questions are expected to be "too hard" for the top/average/bottom team in the field.

This is why I believe that the canon must necessarily be rooted (again, at least at the lowest level) in what is taught at the college level. If we can objectively say the following: if Topic X is taught at a vast majority of schools in lower division courses, then it deserves to be in the "Easy" category; upper division courses in "Regular"; not necessarily taught or taught only at a graduate level constitutes "Hard" - then we have a difficulty scale that measures where the average non-quizbowl-playing person would be likely to get such a question, and we can use this as an objective scale. However, Ryan appears to be using a subjective scale - some unknown function whereby if something shows up a certain number of times it goes from "hard" to "regular" to "easy". In addition, with several categories, especially literature and fine arts, outside of a few things that are universally considered "easy" there's very little ability to distinguish between "regular" and "hard".

So again, we're left with the idea of using ACF tournaments as difficulty "ideals". I'd imagine that every ACF tournament has a target difficulty that more-or-less corresponds to a certain stage in the college curriculum (lower division/upper division/graduate level or not taught) and a certain expectation of how much is going to be converted by the top, middle, and bottom teams in the field. If the ACF Regionals editors feel that last year's tournament was too hard, then they make the tournament easier, and that's our new "Regionals" difficulty that should be emulated at all tournaments of "Regionals" difficulty. If it becomes apparent, after looking at the statistics from all sites, that Regionals was too easy, then the next year's editors should strive to make Regionals slightly harder, and that becomes the new "Regionals" difficulty. I'd imagine that most experienced writers, especially those editing ACF tournaments, think whether a question should be gotten or not when putting it in. Because they don't have any data on whether or not teams actually will get those questions, they have no control over how many teams actually will get those questions.

I guess my question is why this feedback system can't work, or doesn't work, or won't work - ACF tournaments get data from their own tournament and other tournaments of comparable or intermediate difficulty levels, editors decide on a target difficulty for that tournament based on too easy/too hard/just right conversion metrics from the past year, and that becomes the new standard of difficulty that tournaments of comparable difficulty should emulate. In this way we have a very clear concept of "regular" difficulty regardless of how easy or how hard the ACF editors decide their tournaments should be. I, like everyone other than Ryan, am arguing that "regular" should be a subjective term, as it defines the difficulty range of the average college tournament. Perhaps Ryan should use something like "intermediate" to describe his gray area of easier-than-nationals-but-harder-than-novice, because I don't think "Regular" works if it's in the area where Ryan thinks people should get it but 80% of garden-variety tournaments don't use it because it's too hard for their fields.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry
User avatar
Skepticism and Animal Feed
Auron
Posts: 3238
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: Regular difficulty: does it exist?

Post by Skepticism and Animal Feed »

theMoMA wrote:That's not right though. Regular difficulty is written in such a way that a certain percentage of the field can get the tossups at the end, and with specific bonus difficulty in mind. It's not nebulous at all; it's very rigidly defined.
I would like to see data about how good editors are at actually predicting what percentage of the field will get each question in their tournaments.
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source
User avatar
Sima Guang Hater
Auron
Posts: 1957
Joined: Mon Feb 05, 2007 1:43 pm
Location: Nashville, TN

Re: Regular difficulty: does it exist?

Post by Sima Guang Hater »

Ryan, the question you have to ask is, "who are these easy/regular/hard labels are actually for?" Its not productive to have some platonic, pseudo-objective standard of what's "hard" and "easy", simply for "truth-in-naming". Who is this truth for? Its certainly not for people like you, Weiner, and Jerry, who will show up to any tournament that's appropriate for them to play in, regardless of how the difficulty is conveyed. Nor is it really useful for some historical purpose, because we can look at the statistics for a tournament to make judgments about how hard it was for the field, and we can simply look at the packets to see how they stand up to current notions of difficulty.
ryan wrote:I'll give an example. If a tossup on the Bear Flag Revolt goes dead (I'll talk about history since I have a greater feel for that than science), I'm not going to accept the proclamation "that tossup was too hard." Rather, I'm going to say "That's a tossup of reasonable difficulty - but the teams playing this event didn't know it, so it wasn't appropriate for this event." I'm arguing that this is a meaningful distinction.
Why is it meaningful? Isn't it more important that we convey difficulty relative to player strength, rather than to some elite ideal? Quizbowl is expanding, and its more important than we have statements like these
weiner wrote: "regular," defined as "the bulk of tournaments that non-pussy quizbowl teams do in fact play,"
over some fixed ideal.
Skepticism and Animal Feed wrote:I would like to see data about how good editors are at actually predicting what percentage of the field will get each question in their tournaments.
Sure, I can support this.
Eric Mukherjee, MD PhD
Brown 2009, Penn Med 2018
Instructor/Attending Physician/Postdoctoral Fellow, Vanderbilt University Medical Center
Coach, University School of Nashville

“The next generation will always surpass the previous one. It’s one of the never-ending cycles in life.”
Support the Stevens-Johnson Syndrome Foundation
User avatar
grapesmoker
Sin
Posts: 6345
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: Regular difficulty: does it exist?

Post by grapesmoker »

Skepticism and Animal Feed wrote:I would like to see data about how good editors are at actually predicting what percentage of the field will get each question in their tournaments.
Unless you're going to require editors to make that prediction beforehand, I don't know how such a data set would be generated.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
presently: John Jay College Economics
code ape, loud voice, general nuissance
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 6112
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: Regular difficulty: does it exist?

Post by Important Bird Area »

grapesmoker wrote:Unless you're going to require editors to make that prediction beforehand, I don't know how such a data set would be generated.
A weaker form of this would be fairly easy to produce. Looking at past conversion statistics I think it's easy to see what kinds of tossups experienced editors would consider "harder than the rest of this list." I think it's also obvious that editors quite regularly conclude "this tossup is harder than the rest of the tournament, but it's pyramidal and interesting, and it's 24 hours before the tournament is due, so let's just throw it in."
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred
User avatar
Skepticism and Animal Feed
Auron
Posts: 3238
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: Regular difficulty: does it exist?

Post by Skepticism and Animal Feed »

grapesmoker wrote:
Skepticism and Animal Feed wrote:I would like to see data about how good editors are at actually predicting what percentage of the field will get each question in their tournaments.
Unless you're going to require editors to make that prediction beforehand, I don't know how such a data set would be generated.
Since there are now numerical standards for each level of difficulty, you would just have to look at the announcement and see what the advertised level of difficulty was. The harder part would be the modification to stat-keeping; though didn't EFT last year keep track of what bonus parts and TUs were converted?
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source
User avatar
No Rules Westbrook
Auron
Posts: 1238
Joined: Mon Nov 22, 2004 1:04 pm

Re: Regular difficulty: does it exist?

Post by No Rules Westbrook »

Okay, so now Eric is getting into the question of why I like my system, and what its value is.

I'll admit that some of my preference is guttural. I think back to when I was a player just starting out (say, about a year into playing - I'd read some packets here and there and played a few games, but I didn't know much), and I'd go to practice and hear an answer come up that I'd never encountered before - let's say a bonus part on, I don't know, Angelica Kauffmann. I might remark something like "I've never heard of her" and someone would turn to me and say "she's not hard, she comes up." At that time, that remark might have seemed a little odd had I thought about it, but now it seems to make a whole lot of sense. It seems downright intuitive, such that it's exactly what I'd say to someone if today they remarked they'd never heard of Angelica Kauffmann. But, even back then, I think my reaction to someone telling me that was "okay, that means if I do what I need to do in order to become a good quizbowl player - I too will think Angelica Kauffmann isn't hard." And, now that I have more experience, I find that I absolutely agree with that hypothetical statement of my former self.

I mean, everyone takes different paths to getting good at this game, but not that much different. If you want to have competent generalist-level knowledge of some category, like art in this case - there are certain things you just have to learn, like who Angelica Kauffmann is. Competent science generalists need to have an idea of what bremmstrahlung is - it doesn't matter how difficult of a concept it happens to be when you consider it in the context of physics instruction - it comes up in this game, so you have to know it. Sure, you can object that there are players in this game who get good not by studying explicitly for QB, but just by taking classes and being intellectually curious and so on. But, even the best of those players - let's say, a Jon Magin or Andrew Yaphe - have undoubtedly done a lot of work looking at packets and studying stuff that comes up frequently in QB, because you have to do that if you want to be competent and get that answer the next time it comes up. Like Jerry said, this is a game about things that have names - and you have to know stuff about them if you want to be good, regardless of whether or not you learn those things from classes or what.

Plus, it just makes sense to talk to people the way I reference above with Angelica Kauffmann. It's like if you were teaching someone to be a good golfer and they regularly missed four-foot putts - you wouldn't say "well, a high percentage of these duffers out on the course miss those putts quite often, so they're hard." In my opinion, it'd be much more helpful (if you want to create good players) to say to them "look, that's a four-foot straight putt, it's not that hard - you have to be able to make those regularly."

I think my system of labelling gives players an accurate understanding of where they are on the road to becoming a good player - or, even the road to becoming a good player just within a certain subject, like art. It gives them a way more accurate understanding than some definition of difficulty that constantly shifts depending on the intended field.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!
User avatar
fleurdelivre
Tidus
Posts: 535
Joined: Wed May 04, 2005 3:35 am
Location: ???

Re: Regular difficulty: does it exist?

Post by fleurdelivre »

grapesmoker wrote:Unless you're going to require editors to make that prediction beforehand, I don't know how such a data set would be generated.
Well, Magin ranked the Gaddis tossups pre-tournament - if we've still got question-by-question stats, then it would be possible to compare for this one tournament how 3s got answered in comparison to 4s, for example... and this same type of system wouldn't exactly be difficult for other data-geeks to implement in the future if we really want to build such a sample.
Katy
Vanderbilt '06 / Harvard '11 / freelance moderator
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Regular difficulty: does it exist?

Post by Matt Weiner »

grapesmoker wrote:
Skepticism and Animal Feed wrote:I would like to see data about how good editors are at actually predicting what percentage of the field will get each question in their tournaments.
Unless you're going to require editors to make that prediction beforehand, I don't know how such a data set would be generated.
Before Regionals I said I wanted to bring BC up to 15. It ended up being 13.4, so apparently my sense was about 11% off. This is probably as close as most experienced editors come...I doubt NAQT really set out to have a BC of under 14 or have 23 to 25% of their tossups go dead at Sectionals. Everyone always overshoots it on difficulty even when making a concerted effort to bring difficulty down. The lesson here is: if you want BC of 15, shoot for BC of 18, and so on.
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
User avatar
setht
Auron
Posts: 1205
Joined: Mon Oct 18, 2004 2:41 pm
Location: Columbus, Ohio

Re: Regular difficulty: does it exist?

Post by setht »

No Rules Westbrook wrote:I'll admit that some of my preference is guttural.
Comedy gold.
No Rules Westbrook wrote:I think back to when I was a player just starting out (say, about a year into playing - I'd read some packets here and there and played a few games, but I didn't know much), and I'd go to practice and hear an answer come up that I'd never encountered before - let's say a bonus part on, I don't know, Angelica Kauffmann. I might remark something like "I've never heard of her" and someone would turn to me and say "she's not hard, she comes up." At that time, that remark might have seemed a little odd had I thought about it, but now it seems to make a whole lot of sense. It seems downright intuitive, such that it's exactly what I'd say to someone if today they remarked they'd never heard of Angelica Kauffmann. But, even back then, I think my reaction to someone telling me that was "okay, that means if I do what I need to do in order to become a good quizbowl player - I too will think Angelica Kauffmann isn't hard." And, now that I have more experience, I find that I absolutely agree with that hypothetical statement of my former self.
Ryan, I'm very relieved to hear that you don't advocate writing/editing tournaments without reference to the intended audience. To some extent I agree with what you're saying about an absolute scale of difficulty, but I still disagree with your conclusions. In particular, I agree that we could define a scale of difficulty based on how far into a typical college course a particular concept gets covered, or how frequently a topic shows up as an answer (or clue) in various packet archives (although this may not be all that far from relativism--if writers and editors are keeping their audiences in mind when they produce packets, and then you use those packets to define difficulty levels, you're either absorbing the relativist definitions those writers and editors adopted, or you're including packets from a long stretch of time), or using some other basis. I think a "relativist" definition of difficulty is at least as clear as an absolute definition, and I think it's definitely more useful.

When someone turns to you during a quizbowl practice and tells you, "Angelica Kauffmann is not hard, she comes up," you're not interested in finding out that Angelica Kauffmann appears regularly on first-year art history course syllabi, or that Angelica Kauffmann appeared as a tossup answer 18 times in the 1980s--rather, you're interested in finding out that Angelica Kauffmann is a common tossup/bonus answer in the current game of quizbowl, and that she is not considered hard in the current game of quizbowl: given that editors and writers try to write questions that are suitable for a (contemporaneous) target audience, the statement "Angelica Kauffmann is not hard" signals that she's considered fair game in "easy tournaments," which presumably means that many editors and writers believe that lots of current first- and second-year college players will be familiar with her work. I'm sure you and most other long-time players have noticed that some topics cycle in and out of fashion--there'll be a rash of Prester John questions for a while, then that'll die down and some other question fad will come through. During or slightly after the height of "Prester John fever," I bet there were lots of players that suddenly knew more about Prester John than they had prior to the outbreak, which means Prester John became easier in a relativist sense, but presumably not in any sense that is considered fixed and absolute. Let's suppose Prester John was so popular that he was fair game for a novice tournament. Would you say it's more clear and useful to tell a budding novice that Prester John is (current, relativistically-defined) novice-level material, or that Prester John rates 6 out of 10 on the eternal ladder of difficulty, but he's popular right now so there's a decent chance there'll be a question on him even at a novice event featuring mostly questions at levels 1-3 out of 10?

Let's look at tournament announcements. Under the relativist definition, I can post an announcement for ACF Nationals that reads something like this: "This is intended to be a hard tournament, which means I expect it to be challenging to the current top 20 teams (say, target tossup conversion rate of 16/20, and bonus conversion rate of 16 ppb for those 20 teams). The ACF Nationals sets from the past two years should give a fair idea of the target for this tournament." If I write that announcement in an era where Zeke, Subash, and Andrew Yaphe are all top players on different teams, it might be reasonable to expect that the difficulty of the literature questions (in some absolute sense) will be higher than for the science questions. If I write that announcement after they've all retired from national competition, and now 4 of the top teams feature physics/astrophysics grad students, it might be reasonable to expect that the difficulty of the science questions will be higher than for the literature questions. If I try to write an announcement using difficulty terms from an absolute scale, it'll be a mess: "we're shooting for a difficulty of 7 on science, except that we're shooting for an 8 on computer science because we have lots of CS people playing this year, and we're shooting for a 9 on Pomeranchuk cooling because everyone's been studying that non-stop after ACF Nationals 2008..."
No Rules Westbrook wrote:I mean, everyone takes different paths to getting good at this game, but not that much different. If you want to have competent generalist-level knowledge of some category, like art in this case - there are certain things you just have to learn, like who Angelica Kauffmann is. Competent science generalists need to have an idea of what bremmstrahlung is - it doesn't matter how difficult of a concept it happens to be when you consider it in the context of physics instruction - it comes up in this game, so you have to know it. Sure, you can object that there are players in this game who get good not by studying explicitly for QB, but just by taking classes and being intellectually curious and so on. But, even the best of those players - let's say, a Jon Magin or Andrew Yaphe - have undoubtedly done a lot of work looking at packets and studying stuff that comes up frequently in QB, because you have to do that if you want to be competent and get that answer the next time it comes up. Like Jerry said, this is a game about things that have names - and you have to know stuff about them if you want to be good, regardless of whether or not you learn those things from classes or what.

I think my system of labelling gives players an accurate understanding of where they are on the road to becoming a good player - or, even the road to becoming a good player just within a certain subject, like art. It gives them a way more accurate understanding than some definition of difficulty that constantly shifts depending on the intended field.
Ryan, if you're basing your definitions of difficulty and player level on "what comes up in this game" and the writers and editors that decide "what comes up in this game" make that decision based at least in part on their perception of the target audience, it seems to me that your definitions are inherently relativist. They change a bit from tournament to tournament, and they change dramatically over the course of several years. Furthermore, I would argue that this is a good feature: if I want to be a "competent art player," I don't want to study until I reach some fixed and absolute ideal of "competence in the playing of art questions"; I want to get to a point where I can get 20s and 30s on most art bonuses and beat most teams to art tossups, and that depends very much on what my competition (the rest of the target audience at all the tournaments I attend) looks like, both because I have to beat that competition on tossups, and because writers and editors are presumably taking them into account when they're deciding what art questions to produce. I may have to spend more time studying 19th-century French art than I have to spend on ancient, Byzantine and Romanesque art combined, I may need to memorize names and major works for 40 Renaissance artists but only 15 20th century artists, etc. Three years later, that may change and now I need to know more 20th century art than Renaissance art to maintain similar levels of art tossup and bonus conversion.

What say you?

-Seth
Seth Teitler
Formerly UC Berkeley and U. Chicago
President of NAQT
Emeritus member of ACF
User avatar
setht
Auron
Posts: 1205
Joined: Mon Oct 18, 2004 2:41 pm
Location: Columbus, Ohio

Re: Regular difficulty: does it exist?

Post by setht »

While I'm at it, I figured I'd blather a bit about conversion goals vs. difficulty, useful data, and possibly other stuff, since apparently I'm the namesake of this approach.
Skepticism and Animal Feed wrote:
theMoMA wrote:That's not right though. Regular difficulty is written in such a way that a certain percentage of the field can get the tossups at the end, and with specific bonus difficulty in mind. It's not nebulous at all; it's very rigidly defined.
I would like to see data about how good editors are at actually predicting what percentage of the field will get each question in their tournaments.
First off, I think that "conversion goals" by themselves actually aren't a great way to define difficulty, in the following sense: I think pretty much all tournaments would like to see conversion rates of ~15-18 tossups out of 20 and ~15-18 ppb from the target audience. Does anyone ever want to see tossup conversion less than 75%, or bonus conversion less than 50% (and does anyone ever want to see 100% conversion on everything)? I think the real difference separating "easy," "regular" and "hard" is in the target audiences for events at those different levels.

Second, a conversion goal of 16 tossups out of 20 doesn't mean that the writers and editors are trying to get the set to a point where every tossup is answered in 80% of the rooms--there's some variation from tossup to tossup, and I think it's fine for the editors to leave in tossups that they feel certain are going to be converted at somewhat higher or lower rates.
The New SCIENCE! wrote:Well, Magin ranked the Gaddis tossups pre-tournament - if we've still got question-by-question stats, then it would be possible to compare for this one tournament how 3s got answered in comparison to 4s, for example... and this same type of system wouldn't exactly be difficult for other data-geeks to implement in the future if we really want to build such a sample.
Third, setting goals for tossup difficulty entirely in terms of conversion rates is kinda weak--after all, a tossup with 11 lines of impossible clues followed by a butt giveaway will have the same conversion rate as a speedcheck consisting solely of the butt giveaway, and could have the same conversion rate as a good, pyramidal question. It would be better to have more detailed goals in mind, but we don't keep detailed stats on where people buzz on tossups so we don't really have any good way of telling which tossups had useless lead-ins, which tossups had giveaways in the middle, etc. I'm not sure how Jonathan produced his Gaddis tossup rankings, but if he looked at anything besides the difficulty of the answer and the giveaway (for instance, the difficulty of the lead-in, or the difficulty of the 4th clue), his rankings probably won't be predictive of conversion rates.
Matt Weiner wrote:Before Regionals I said I wanted to bring BC up to 15. It ended up being 13.4, so apparently my sense was about 11% off. This is probably as close as most experienced editors come...I doubt NAQT really set out to have a BC of under 14 or have 23 to 25% of their tossups go dead at Sectionals. Everyone always overshoots it on difficulty even when making a concerted effort to bring difficulty down. The lesson here is: if you want BC of 15, shoot for BC of 18, and so on.
This is a nice example of an experienced writer and editor incorporating empirical data from a previous tournament in an effort to improve future tournament sets. I think it's also an example of how we could do an even better job improving as writers and editors (if we had time and inclination) if we kept more detailed statistics: presumably ACF Regionals 2008 had some variation in bonus difficulty. If Matt Weiner makes all bonuses slightly easier, he can hit his goal of 15 ppb, but he probably won't improve on bonus variability. If we kept track of which bonus parts were answered, he could get a better feel for which bonuses were too hard or too easy--perhaps the geography bonuses were systematically easier than the art bonuses--then work on those, leaving untouched those bonus types that hit his goal. Presumably he could get closer to his conversion goal and reduce variability at the same time if he had that data.

-Seth
Seth Teitler
Formerly UC Berkeley and U. Chicago
President of NAQT
Emeritus member of ACF
User avatar
ValenciaQBowl
Auron
Posts: 2558
Joined: Thu Feb 05, 2004 2:25 pm
Location: Orlando, Florida

Re: Regular difficulty: does it exist?

Post by ValenciaQBowl »

"Angelica Kauffmann is not hard"? "She comes up"???

I just got done googling this "Swiss Neoclassical painter" of the 18th century, so I guess now I've heard of her. I don't mean to sidetrack the overall discussion, but I have a minor in art history, 12 hours of upper-division classes plus plenty of lower-division surveys, and this person is new to me. I realize that this certainly may be the fault of my own instructors' choices in content and/or my own sloth in not doing independent work on female 19th-century artists, but I haven't seen her in packets, either, though I'm sure that if y'all are saying she's come up, she has and it's my problem for not noticing. And of course, I'm not a regular circuit player, so presumably I've missed references to her.

I suppose my surprise at hearing that this person is not considered "hard" relates to this discussion only in this: is it fair to say that Ms. Kauffmann would not be considered a "regular difficulty" answer? Or is the consensus that she would be fair game for a "novice" tournament?
Chris Borglum
Valencia College Grand Poobah
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Regular difficulty: does it exist?

Post by Mechanical Beasts »

I searched "angelica kaufmann" site:quizbowlpackets.com, and got the following:
2005 ACF Regionals wrote: 9. Name these 18th-century painters, FTPE:
A. Though born in Switzerland, this artist’s best works were produced in England. Those works include 1783’s Telemachus and the Nymphs of Calypso.
Answer: Angelica Kaufmann
B. This British artist was despised by Blake, who found his Discourses insipid. His paintings include Lady Sarah Bunbury Sacrificing to the Graces.
Answer: Sir Joshua Reynolds
C. Though born in America, this painter became president of the Royal Academy. He’s best known for a 1770 painting that depicts the death of a British general.
Answer: Benjamin West
1998 Terrapin Invitational wrote: 8. Artistic talent sometimes runs in families. On a 5-10-15 basis, give the surnames of these artistic brothers.
5. Pol, Hennequin, and Herman illustrated the Tres Riches Heures du Duc de Berry, completed in 1416.
Answer: Limbourg
10. James and Robert were successful architects in the neoclassical and rococo styles. London’s leading interior
decorators of the later eighteenth century, they often worked in collaboration with Angelica Kaufmann.
Answer: Adam
15. Antoine and Mathieu were successful painters but less famous than their brother Louis, who painted
Peasant Family around 1640.
Answer: Le Nain
So, in all, the hard part (I think) of a Regionals bonus, and a mention in a bonus part. Make of that what you will.


EDIT: and hey, if I had spelled her name right, I would have gotten mentions from a leadin (Regionals 2006) on Johann Joachim Winckelmann, a tossup from the second finals packet of Pontormo, and a bonus part from the 2001 "Antonin Artaud Tournament of Cruelty."

EDIT: on Stanford, there's also a result for a BOB 2005 bonus part, a mention in a bonus part on Neoclassicism at 2003 Wildcat, a 2003 CO bonus part, and a bonus part (off the clues "Self-Portrait - Swiss - 1741-1807") at 1996 Nats. So she comes up. She's certainly no tossup answer at anything but a high level.
Andrew Watkins
User avatar
ValenciaQBowl
Auron
Posts: 2558
Joined: Thu Feb 05, 2004 2:25 pm
Location: Orlando, Florida

Re: Regular difficulty: does it exist?

Post by ValenciaQBowl »

Thanks for doing the leg work, Andrew. She does come up, but I'd still say she's hard.

Sorry for the interruption: back to the discussion of regular difficulty.
Chris Borglum
Valencia College Grand Poobah
User avatar
Skepticism and Animal Feed
Auron
Posts: 3238
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: Regular difficulty: does it exist?

Post by Skepticism and Animal Feed »

I think it would be degenerate of us to go into the validity of Ryan's examples rather than to focus on the larger point he uses them to make.

If Angelica Kaufmann did not exist, it would be necessary to invent her.
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Regular difficulty: does it exist?

Post by Mechanical Beasts »

The tangent to the regularly scheduled argument was interesting, at least, and hell, if it allows me to buzz on a tossup with heavy "Swiss woman" implications, it's ten minutes well spent.

Let us please now have a deathmatch between "tossupability of obscure characters from Candide" vs. Angelica Kauffmann.
Andrew Watkins
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Regular difficulty: does it exist?

Post by theMoMA »

everyday847 wrote:The tangent to the regularly scheduled argument was interesting, at least, and hell, if it allows me to buzz on a tossup with heavy "Swiss woman" implications, it's ten minutes well spent.

Let us please now have a deathmatch between "tossupability of obscure characters from Candide" vs. Angelica Kauffmann.
There are obscure characters in Candide?
Andrew Hart
Minnesota alum
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Regular difficulty: does it exist?

Post by Mechanical Beasts »

theMoMA wrote:
everyday847 wrote:The tangent to the regularly scheduled argument was interesting, at least, and hell, if it allows me to buzz on a tossup with heavy "Swiss woman" implications, it's ten minutes well spent.

Let us please now have a deathmatch between "tossupability of obscure characters from Candide" vs. Angelica Kauffmann.
There are obscure characters in Candide?
I suppose not really. There are certainly characters about whom there's so little material that they're not tossupable: a Brother Giroflee tossup, or a tossup on that commandant, or on the king of El Dorado, just wouldn't be fun to write. But none of them, like, aren't known.
Andrew Watkins
User avatar
No Rules Westbrook
Auron
Posts: 1238
Joined: Mon Nov 22, 2004 1:04 pm

Re: Regular difficulty: does it exist?

Post by No Rules Westbrook »

Hmmm, maybe she wasn't the best example - though, come on, I know there are some Angelica Kauffmann lovers out there, make your voice heard!

Anyway, let me try to respond to Seth's post. I didn't mean to suggest that there's some "eternal ladder of difficulty" - just because I don't think that difficulty-labelling should be tied to field strength doesn't mean that I think it can't change at all over time. I think difficulty-labelling should be tied to the reasoned opinions of experienced people currently involved in the game (whether by writing, editing, playing) - because those players are the ones with a feel for the current canon.

I don't think there's as much of a "Prester John problem" as you indicate. The thing is, I think - the canon, as it stands in the minds of many of the elite players, tends to get larger and larger - it rarely ever gets smaller, even if we try to make tournaments easier. Even though the past year has seen something of a cooling-off period in difficulty, I think the official canon is still getting bigger and bigger. Even though there may be some things that don't come up as much as they used to, many of the experienced players still know those things. So, I think when you ask the current experienced players, you get a fairly accurate reflection of the current difficulty of a given answer - of course, if that answer comes up 12 times in the next six months, the answer you get from them will change so as to indicate that the answer is now easier. We do that all the time when we write to expand the canon - we consider "well, I'd never have written a tu on this two years ago, but now it seems about time that it come up."

And, let me suggest this (kind of out there) theory. When I say to a novice that "Answer X isn't that hard" - my thesis is that I'm telling them "when you become a good generalist-level player, you too will think that Answer X isn't that hard." Note - My key assumption in making a statement like this is that any player wants to become a good generalist-level player (at least in a certain subject). My assumption is not that they just want to become "good enough to win" - hell, if they're playing in some circuit like the South (no offense, the South) - you really don't need to be that good to win. Rather, I'm saying something like "If you want to be thought of as a good player (at least in this subject), you shouldn't think Answer X is that hard." This kind of fits in with my general theory that the whole point of playing QB is "playing the game well," not necessarily winning or scoring a lot of points or getting trophies or stuff like that. I've advanced the notion that the goal of this game is simply to "play well."

Wow, this has all gotten way more theoretical than I ever intended, to the point that it's surely not making much practical sense anymore. But, I hope that maybe I've persuaded at least some people that my position isn't absolutely nuts. And, also that my goal here isn't to lobby for more hard questions or to say that we should ignore field strength in writing or anything like that.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!
User avatar
alkrav112
Rikku
Posts: 269
Joined: Sun Dec 26, 2004 8:55 pm
Location: Ann Arbor, MI

Re: Regular difficulty: does it exist?

Post by alkrav112 »

No Rules Westbrook wrote:Hmmm, maybe she wasn't the best example - though, come on, I know there are some Angelica Kauffmann lovers out there, make your voice heard!
I'm the guy.
Andy Kravis
Michigan '09
Columbia Law, '13
Locked