Clues that reward understanding

[magin]

I decided to start this thread after reading the high school posts about the GP equation (I have also buzzed on it as a clue and received points, despite knowing utterly nothing about it besides mere word association). I contend that quizbowl clues should attempt to reward understanding of the answer rather than word associations, and that currently, many clues allow players who only know word associations (as in, say, the Huang-Minlon modification being associated with the Wolff-Kischner) to beat players who may understand the answer much better.

Does this mean that buzzing off word associations is bad, or should never be rewarded? I don't think so; due to the nature of quizbowl, everyone accumulates some amount of clues through "quizbowl osmosis" (which I define as hearing a clue in a question and subsequently associating it with an answer), and that's fine. However, I think that clues rewarding such osmotic knowledge are not as good as clues that reward understanding of the answer.

Practically, then, what kind of clues reward understanding? I don't know if I have the answer, but I have some ideas. I think that a good way to begin looking for clues is to ask yourself, when writing a tossup, "what would people who understands this answer or this clue know? How can I word it in a way that will be clear to them?" For instance, when I write a tossup on a poet, I read some of his or her poems that I think are notable, and try to understand the arguments of the poems. Then, I try to incorporate lines from the poem that someone would recognize and understand if he or she had some sort of basic understanding of the poem.

Here's an example: Milton's sonnet "When I consider how my light is spent," superficially, is about his blindness, so you could write: This author described his feelings of blindness in his sonnet "When I consider how my light is spent." However, I don't think that would do a good job of allowing someone who understands the sonnet to buzz before the title. You could say: This author described spending "half [his] days in this dark world and wide" in his sonnet "When I consider how my light is spent." That's definitely an improvement, but if you understand the argument of the sonnet, then that line isn't really key to understanding what's going on. Here are some ways I might incorporate a basic understanding of the sonnet into a concise clue:

This author compared his literary gift to the parable of the talents in a sonnet lamenting "that one talent which is death to hide / Lodg'd with me useless." [The parable of the talents is an important image in this poem. Here, I tried to describe its place in the poem.]

This author asked "Doth God exact day-labour, light denied?" and hears Patience reply "They also serve who only stand and wait" in his sonnet "When I consider how my light is spent." [This is a concise form of the main poetic argument of Milton's sonnet.]

In my opinion, the two preceding sentences are much better than [This author described his feelings of blindness in his sonnet "When I consider how my light is spent."] and [This author described spending "half [his] days in this dark world and wide" in his sonnet "When I consider how my light is spent."], since they are better at describing what's actually going on in the poem, and would reward someone who has read and understood the sonnet.

This principle can be applied to other fields, not just literature; I would certainly welcome more clues that involve knowing important scientific concepts rather than knowing associations of words. What do you all think?

[Skepticism and Animal Feed]

I kind of find it annoying when tossups about a text are full of one-line quotes from a text, or, even more, when tossups about a historical event are full of quotes about that event. Recent examples might include the "Book of Certitude" TU from MO.

Sure, if there was a Baha'i player who memorized every line of the Book of Certitude, he would get it. But I feel that TU's time would have been better spent summarizing important arguments in the book rather than just giving us a half-dozen quotes.

[No Rules Westbrook]

A few practical observations:

everyone accumulates some amount of clues through "quizbowl osmosis"

This is, conveniently I think, a massive understatement. People accumulate a ton of clues through "osmosis" and generally studying qb. Any way you slice it, this is an enormous part of the game - even if you're a Maginesque-type player.


Secondly, we all agree that putting clues before a title is useless if those clues aren't important or recognizable, or they don't allow you to buzz. Noone can buzz on "described his feelings on blindness" unless they randomly guess - if you give that kind of clue, you may as well just give the title cause it's the only buzzable thing.

I certainly agree that your two preferred examples are very well done, but I'd note that they're just as subject to osmosis buzzing as anything else. I can memorize the "parable of the talents" just like I can memorize "When I consider how my light is spent" - I can also remember those lines of poetry you quote. The general need to be concise makes qb a game about "chunks of knowledge" - but that knowledge can be acquired in lots of ways - ways you'd probably think of as legitimate and scholarly, and ways you'd probably think of as cheap osmosis.

[grapesmoker]

Let me share my views on this matter with regards to science (or, at least, physics and math).

I've taken a lot of physics and math classes at this point, and I can tell you for sure that the majority of things that are learned in those classes are not named associations like Gross-Pitaevskii = BEC. I mean, if a question told you, "These phenomena are described by a nonlinear version of the Schrodinger equation with an operator that includes a term proportional to the square of the wavefunction magnitude," most non-physicists would have no idea what that means (unless they read up on it and understood something about the math involved). Of course, I'm describing the GP equation for the purpose of explaining the difference between understanding what such a thing is and simply associating two things with each other.

In general, I think quizbowl science has gone overboard in two areas. First, the drive to find more difficult clues, ostensibly to challenge better players, has led to the unearthing of innumerable second-order effects, associated eponymous constants, and other such mostly useless information that very few people (and virtually no one in quizbowl) actually know anything about. Correspondingly, these things filter through the clue space and into bonus parts and even tossup answers for no reason other than that they have come up before, causing people to write on them again. The second problem is that the drive to write questions like the above has displaced understanding the phenomena at hand in the game and simply caters to people who memorize lots of packets or read Wikipedia articles. This was brought home to me in dramatic fashion when, during the MO final, Eric beat me to a tossup on inflation. Now, Eric is a phenomenal science player who knows a great deal of stuff, so I don't mind losing a buzzer race to him, but in this case it was particularly weird because I actually study inflationary physics. I'm no great shakes as an inflation theorist, but I bet I could tell you more about the topic than almost anyone else in the game right now, and yet here I am not getting questions because I essentially failed to figure out which aspect of Dirac-Born-Infeld theory the question is talking about. I guess one answer to my complaint might be, "learn more theories of inflation, chump," but I think this is not the right answer. While there are a great number of string theorists outside of quizbowl concerned with building string-theoretic models of inflation, many of these models do not have particularly widespread exposure and there's no reason for anyone not directly working on them to know what they are. Not to mention that if you really want to see if someone knows that model, you should actually describe it (see the GP discussion above) whereas just giving the model's name is benefiting someone who just read Wikipedia's "List of inflationary models," category more than it's benefiting someone who actually works in cosmology.

I don't want to create the impression that MO was particularly atrocious in this regard; it had its share of very good questions (as an example, I offer the bonus on Fermi's golden rule, which actually contained 3 things you are very likely to study if you have taken upper-level QM), and it's not particularly different from many other tournaments with respect to these problems. But I think the persistence of these problems is detrimental to quizbowl; it artificially drives up the difficulty for people whose knowledge doesn't come from packet perusal and it also eliminates advantages that actual scientists gain from actually studying the material. These are both bad things and their progress should be checked. In general, I would like to see more questions that reward actually having done the work on some particular aspect of science rather than memorizing a bunch of associated terms. It actually seems like the chemistry and bio questions tend to do a decent job of rewarding that (this is my outsider's perspective) but for some reason a lot of the physics and math writing has failed to adopt this model, and that really needs to change. When I was chatting with Mike Sorice about this in IRC last night, he pointed out that even if a question perhaps generates a buzzer race between two really knowledgeable physics players but not anyone else, that's not actually a problem with the question; people with lots of physics knowledge should be able to buzz well before anyone else.

I would encourage people to do the same thing at the college level that I suggested they do at the high school level: write on topics of basic general interest and use deep clues, rather than writing about stuff like the GP equation, which hardly anyone is going to get. To be sure, at harder tournaments, we should be free to explore more of the canon space, but there are so many important things that require deep knowledge and yet are routinely ignored by writers that I don't think we're in danger of exhausting difficult, interesting topics any time soon. Of course, this requires that people pay more attention to textbooks and class notes than to packet archives and old Wikipedia articles, but I think it's a worthy, necessary goal and will result in a more rewarding playing and writing experience for all concerned.

[grapesmoker]

Secondly, we all agree that putting clues before a title is useless if those clues aren't important or recognizable, or they don't allow you to buzz. Noone can buzz on "described his feelings on blindness" unless they randomly guess - if you give that kind of clue, you may as well just give the title cause it's the only buzzable thing.

If I heard something about a blind poet, I sure would be tempted to buzz with Milton. People can and do buzz on that.

I certainly agree that your two preferred examples are very well done, but I'd note that they're just as subject to osmosis buzzing as anything else. I can memorize the "parable of the talents" just like I can memorize "When I consider how my light is spent" - I can also remember those lines of poetry you quote. The general need to be concise makes qb a game about "chunks of knowledge" - but that knowledge can be acquired in lots of ways - ways you'd probably think of as legitimate and scholarly, and ways you'd probably think of as cheap osmosis.

Look, quizbowl is not played by robots who memorize everything. Yeah, you can "memorize" the parable of the talents, but at some point, I'm happy saying that if you know that interpretation, it's not really memory at all and that you have some understanding of the work. On philosophy questions, I often make good buzzes on things that I haven't read; I am able to do this because I understand the underlying ideas and am able to relate them to specific arguments put forth by specific thinkers. Is that fraud? I don't think it is.

Of course at some point this game is about memory. We all acknowledge that. But when you study something, you come to remember a lot of things about that thing, and we should structure our questions in such a way as to reward that, rather than do things like throw out a million associations or random titles.

[Cheynem]

Yeah, I agree with Jerry that there is a difference in terms of "memorizing" and just making word associations based on clues. Quiz bowl is about memory, every buzz is based on something remembered. The key difference is in what sort of "memorizing" is being rewarded. I don't really know much about Scylla, but she came up when I was researching a tossup on Minos I was writing, so that's why I could power that tossup. I think that sort of knowledge should be rewarded--I was doing good-faith research and found some clues. On the other hand, I've never read "The Idea of Order at Key West" and don't know much about it, so I couldn't buzz until I heard "Ramon Fernandez." That's what makes that a buzzword clue and why the clue was appropriately placed near the end of the question.

[No Rules Westbrook]

Okay, science is the one great exception in this discussion. It's tough to "memorize" or "osmose" things about science because you can formulate and restate equations in different ways, you can throw in a lot of jargon that's difficult for non-science people to correctly interpret, etc.

Let's put science in a separate category when we start talking about what kind of knowledge we want to reward - because science is different - it's a subject where academic curriculum and study plays a way different role than in any other qb topic.

Science aside - My point in this thread is that trying to distinguish between "buzzword knowledge" and "osmosed knowledge" and "qb study/writing knowledge" and "knowledge gained through good-faith study" and "knowledge gained through academic study" and whatever else is very difficult - trying to separate one of those types of knowledge from the others is a futile effort. Any clue can be a "buzzword," no matter how important it is, and no matter how rewarding of understanding it is.

Noone disagrees that we should always try to choose legitimately important clues, and that we should use academic study and "what people in the field know" as criterion by which to gauge importance. Noone wants the clues and answers that come up in QB to have no external importance, to just be a series of random insular assocaitions. I don't know why people always act like I think this. All good writers, notably me for one, try to pack their questions with clues that reward deep knowledge, especially early in the question.

Packets today, when they're written well, are as fraud-proof as they're gonna get - but you gotta realize, that's still not very fraud-proof! (note: "fraud" is a word that gets applied to lots of knowledge wrongly, but I'm just using it as a byword). I do think the Magins of the qb world have sugar plum visions or aspirations that this game can turn into something it's just not - it's not a reading circle or an essay contest, "understanding" is never gonna be at the premium you want.

[setht]

I think one thing people can do is make more of an effort to start out by picking answers that aren't over-exposed. If I pick a reasonably important topic that hasn't been hot in recent quizbowl, I can write pretty straightforward clues without having to worry too much about word-association type buzzes. Here's a science tossup I wrote for Minnesota Open:

Atoms of this element are used to create artificial “guide stars” in the upper mesosphere through resonant backscattering of light from dye lasers tuned to 589 nanometers in certain adaptive optics systems. Such lasers interact with atoms of this element to excite one spectral line in a doublet whose broadening upon application of a magnetic field was the first known example of the Zeeman effect. That doublet corresponds with the Fraunhofer D lines and its strength is exploited in the design of low-pressure vapor lamps. Aldosterone promotes kidney reabsorption of ions of this element, which is actively transported out across plasma membranes in a mechanism used in establishing resting potentials. FTP name this element whose ions are moved by a pump that also acts on potassium and whose strong doublet produces a yellow light in flame tests.
ANSWER: sodium or Na

I think it wasn't used, which is a shame because I would have liked to see how it played. All the non-biology clues refer in some way to the sodium doublet and its importance in various fields. I don't think this is something that has come up a lot in recent quizbowl, and the question definitely doesn't have any "fill-in-the-blank in these multiply-eponymous things" clues or lots of semi-connected name-dropping clues a la quizbowl organic chemistry. In any case, adaptive optics is important stuff but doesn't come up; since it doesn't come up, I can go ahead and give a very straightforward clue in the first sentence that I think anyone with knowledge of AO could buzz on.

Identifying under-exposed areas to mine for questions is not easy, but I think Jerry's advice for science and math is a decent guide for other areas as well: look for basic, important topics and then research deep clues, which I interpret as a suggestion that people write more tossups on "core" answers (adopting the terminology introduced by Andrew in this other thread) and cut back a bit on the number of tossups on peripheral/random outlier answers they submit for hard tournaments. If this means fewer Edwidge Danticat questions and more questions on Keats's oeuvre I think that's for the best. I personally think that the goal of rewarding understanding might be easier to achieve if (say) the literature distribution shifted a bit more towards US/British (and maybe European) literature and away from world literature, but even within the confines of the current distribution I think writers could do a better job of picking under-exposed, worthwhile answers, which then makes it relatively easy to use clues that reward understanding.

-Seth

[grapesmoker]

Science aside - My point in this thread is that trying to distinguish between "buzzword knowledge" and "osmosed knowledge" and "qb study/writing knowledge" and "knowledge gained through good-faith study" and "knowledge gained through academic study" and whatever else is very difficult - trying to separate one of those types of knowledge from the others is a futile effort. Any clue can be a "buzzword," no matter how important it is, and no matter how rewarding of understanding it is.

At some point it becomes hard to tell the difference between "knowing how" and "knowing that." Again: if someone literally memorizes the entire plot of The Mill on the Floss they're going to buzz on the first clue of that question every time. I don't care if that happens because there's no possible way for someone to do this with, say, all literature. The answer space is just so varied that some hypothetical player who simply wanted to memorize all the details about all literary works would be busy forever and a day.

Packets today, when they're written well, are as fraud-proof as they're gonna get - but you gotta realize, that's still not very fraud-proof! (note: "fraud" is a word that gets applied to lots of knowledge wrongly, but I'm just using it as a byword). I do think the Magins of the qb world have sugar plum visions or aspirations that this game can turn into something it's just not - it's not a reading circle or an essay contest, "understanding" is never gonna be at the premium you want.

I don't think that's true. I have played questions in recent memory that did a poor job of rewarding you for having read something, for example. I'm actually worried less about the kind of thing that Jonathan is talking about, where some clue may be phrased sub-optimally, than I am about situations where, for example, a tossup on a work I've read contains 5 lines of references to minor incidents without a single character name in sight. Unless the incident is a pivotal one, it's going to be hard for most people who have read that work to buzz there, which is fine for one or two clues, but not for half the tossup. As I say above, I'm not sure that we need to worry about someone memorizing every minor character from whatever book that we're basically punishing people who have actually read it by making the question unanswerable.

[grapesmoker]

Seth posted while I was writing my previous post, so I just wanted to say that this sodium question is a good example of the kind of thing I'm talking about. If you know stuff about adaptive optics, you get 15 points; I don't, so I'd probably be buzzing somewhere around the Zeeman and Fraunhoffer clues, but either way, I think you're going to reward understanding and real knowledge much better if you write questions in this fashion.

[cvdwightw]

The problem with Jonathan's assumption about "rewarding understanding" is - as Ryan points out - over time, any good clue will just become another clue to be associated with the answer. Sure, it will work wonders the first few times it's done, but then people will continue to use it (because it's important, it comes up in lots of textbooks, etc. - this is just a rosy picture of "people trying to write questions without resorting to packet archives for clues and answers," but acknowledge that it also works with "it's come up before, so it's okay to use it again").

I think that there's something else to take from Jonathan's argument: if people are going to do this memorization thing, they might as well be memorizing clues that are useful to people who actually study the thing. For instance, in science, knowing what an equation says and how to calculate things from the equation is almost always more important than the name of that equation. In literature, specific lines/themes/actions that are critically studied are always more important than random lines/etc. that don't get a lot of play. Knowing the significance of the result of a battle is almost always more important than knowing what happened during the battle, which is almost always more important than just knowing the names of people involved. The point here is that it's virtually impossible to stock a question with good clues that always reward understanding over quizbowl osmosis; however, we can make an effort to stock a question with clues in which quizbowl osmosis points the player in the right direction of understanding. To re-use Jerry's example, quizbowl osmosis of what the G-P equation is will provide a player (assuming that he's quizbowl-osmosed other clues like "Schrodinger equation") with better knowledge of what BECs are and why they're important than merely knowing the name of the equation.

[setht]

Packets today, when they're written well, are as fraud-proof as they're gonna get - but you gotta realize, that's still not very fraud-proof! (note: "fraud" is a word that gets applied to lots of knowledge wrongly, but I'm just using it as a byword). I do think the Magins of the qb world have sugar plum visions or aspirations that this game can turn into something it's just not - it's not a reading circle or an essay contest, "understanding" is never gonna be at the premium you want.

I don't think that's true. I have played questions in recent memory that did a poor job of rewarding you for having read something, for example. I'm actually worried less about the kind of thing that Jonathan is talking about, where some clue may be phrased sub-optimally, than I am about situations where, for example, a tossup on a work I've read contains 5 lines of references to minor incidents without a single character name in sight. Unless the incident is a pivotal one, it's going to be hard for most people who have read that work to buzz there, which is fine for one or two clues, but not for half the tossup. As I say above, I'm not sure that we need to worry about someone memorizing every minor character from whatever book that we're basically punishing people who have actually read it by making the question unanswerable.

I think this is an issue of obfuscation run amok, which is certainly one of the more pernicious quizbowl writing trends of recent memory. I think it might be worthwhile at some point to start a separate thread on vagueness/obfuscation in quizbowl questions, since I think Jonathan and others have been primarily focusing on the somewhat different issue of finding clues that are preferentially going to generate "real knowledge" buzzes vs. "word association/packet knowledge" buzzes*.

* The more I think about this, the more I feel that the points Ryan and Dwight have made indicate that the way to do this is to find clues that haven't come up a lot before. In order to find enough such clues to stock most/all of a tossup, one method is to find topics that haven't been asked about a lot in recent memory. Another option is to go with a "core" answer that has been asked about recently but has so many worthwhile clues available that a mostly- or entirely-new tossup can be written. Piggybacking on Jerry's earlier bit about the Gross-Pitaevskii equation and Dwight's more recent post, finding a new clue doesn't have to mean finding some material that has not come up at all--if all the BEC questions in recent memory just name-drop the GP equation without describing what it says, a writer could generate a fine new clue by describing the content of the GP equation without giving the name.

-Seth

[magin]

I think Ryan and Dwight aren't understanding my argument, and are perhaps making reflex arguments against it. I fully agree that many buzzes are based on "quizbowl osmosis" and word associations. However, we should strive to write questions to reward understanding, not such word associations; that's a separate issue from people buzzing in off word associations (which is probably inevitable in our game).

Furthermore, if we do our best to write questions that reward this understanding, perhaps players would attempt to understand more answers than merely learning such word-associations. Currently, going out and learning many one-to-one word associations is fairly rewarding to players, and also easier than attempting to understand answers. I'd be more satisfied with a game that made more of an effort to reward understanding first and word associations second, and I think it would be beneficial to good writing practices to discuss ways of writing questions that reward this understanding.

[Captain Sinico]

Of course Ryan and Dwight are right that people will memorize clues, often without even really trying. So, if we want questions that do a better job deconvolving "knows only due to memorizing old packets" from "knows," it's necessary that we not re-use clues at the levels we currently do. So, writers will then have to go find some new clues, or find new ways to state them. To take up an example already multiply posited*, one might give the form of the Gross-Piatevskii equation but not its name.
It's true, of course, that we can't ever get away from rewarding "fraud" however defined at some level; that's actually a good thing, in my view, because even fraudulent associations are knowledge of a (not very deep) kind. I say, however, that "fraud" should be rewarded less strongly than it is now. That's not an impossible end by any means.

MaS

*Sorry to piggyback on everyone else here if doing so seems repetitive to the reader.

[naturalistic phallacy]

At some point it becomes hard to tell the difference between "knowing how" and "knowing that."

Jerry, you are my hero.

In all seriousness, this was the point that I was attempting to get across in the IRC discussion. It's often very difficult to make a clear distinction between a player who studies and the player who memorizes, especially as questions attempt to improve to reward the former.

[No Rules Westbrook]

Okay, this'll probably be a long and theoretical post. I was going to make it a new topic, but it fits well in this discussion, responding to Jerry and ohters. Here goes, it's titled:

The Rationality of the Robot Gambit

This is a thought experiment that imagines a robot playing qb - the robot's knowledge stems only from previous packets, and it has a great memory (not a perfect memory, just very very good - because I want the robot to be a potentially realistic player). Now, no real player in QB is even close to a robot nor could they be - being involved in qb means at the very least that you probably accumulate knowledge through writing, you gain experience about how to play by being in matches and hearing questions, you surely have some knowledge from personal likes/dislikes, and almost every solid qb player is a fairly smart person with academic interests, etc.

But, let us imagine a simple robot, to make this easy. I contend that learning to play like the robot is very rational. No matter how you write the questions, no matter how hard you try to reward understanding and thwart the robot - the robot is going to munch you up and spit you out, and there's nothing you can do - just like that robot in SkiFree.

Here's why. The robot concedes you will beat him if you can buzz on any clue that hasn't come up before. So, sure, there are always more clues out there that the robot doesn't know - there are more characters, more arguably important theoretical interpretations, more lines of poetry, and so on. But, the robot doesn't care unless you can buzz on those clues - the robot doesn't have to memorize very minor characters and individual lines of text because, hey, unless you just read that book yesterday - you're probably not gonna be able to buzz on those things either, no matter how legit your knowledge is! So, the robot is very content to just let those clues pass and wait for one he can buzz on.

Even if you can buzz on a few of the above kinds of clues, because you have super-deep knowledge of a few tossup subjects, the robot bets that you can't do it nearly enough times in a match to be able to beat him. Maybe you can do it if you assemble a group of three or four specialists, but otherwise, it probably ain't happening.

Contrary to what Jerry says, the robot has time on his side. In order to learn about The Mill on the Floss, you need to pick up the book and read it - that takes time. The robot need only comb packets and memorize things that have come up before about The Mill on the Floss. The robot can accumulate info at a much faster rate than you can.

Further, the robot gets stronger as more packets are produced, and as difficulty goes up. Like Dwight says, those things that used to be "golden clues" - i.e. clues that people with reasonably deep real knowledge know but haven't come up before - after those clues are dropped in a packet, they're just another clue that the robot will munch up. There's only so many "golden clues" in the world - and with several subjects in qb, they're very limited.

Now, like I said above, science is probably an exception - because you can reformulate equations and so on. Maybe there are a few other instances where you can sufficiently confuse the robot so that he's unable to read the clue, but there aren't nearly enough. Seth is right that it helps to search for things that haven't come up lately, but it's my impression that most writers always do that (I know I do).


DISCLAIMER:
The point of this post is not to encourage anyone to play like a robot. Rather, I contend that all of us play like robots to varying degrees at certain times and in certain subjects - it's a necessary consequence of playing qb. The point of this thought experiment is to demonstrate that the robot can't be defeated, so formulating a framework antagonistic to him is futile and misguided.

Rather, we should just continue to do what most good writers already do - pack questions with important relevant clues, and make sure that both our answers and clues have independent justification. Don't bother trying to fight the robot...get out SkiFree again if you don't believe me.

[grapesmoker]

I think the important question here that is surely on all of our minds is whether the robot has phenomenal consciousness. For ten points, write a 2,000 word post defending either position.

[Strongside]

Ryan's post about the robot, and his other posts in this thread are excellent, and I agree with them.

As for robots, I think it would be cool to create of someone could create both a quiz bowl playing robot, and a quiz bowl question writing robot.

[magin]

Ryan, I don't think your robot argument really belongs in this thread. If people want to play like robots, that's fine with me; I personally gain more satisfaction from buzzing off understanding an answer than buzzing like a robot, but that's my choice, and I'm fine if people get the most satisfaction from playing like a robot.

However, this isn't about "beating the robot." This thread isn't about denouncing robotic buzzing (which is part of the game, and isn't problematic in itself). Instead, it's about attempting to write questions that reward understanding more than robotic play. In my opinion, questions that reward robotic play over understanding are problematic; do you agree? Do you think that writing questions with the aim of allowing players who understand a topic well to buzz before robots is a good aim?

[No Rules Westbrook]

Yeah, I think most writers already try their best to achieve that aim though. If they don't, your post about how to leadin to titles with good concise/important information is useful.

[cvdwightw]

I think it's been hashed out over and over again in this thread that we should write questions that reward understanding over rote memorization. Jonathan, I don't think you'll find that anyone in this thread, even those that disagree with what you're saying, disagrees with that basic premise on which you've constructed this thread.

I'll have a much longer post out in a while, but Ryan's Rational Robot is right on. Ryan is simply pointing out that fraudulent knowledge will always, in the long run, beat out "real" knowledge. The question that you pose, then, is how to write questions such that the short-term gap between these two forms of knowledge is minimized, thus making the "long run" time scale significantly longer than 20 questions.

This whole discussion reminds me of Bruce's Unfalsifiability of Quizbowl Claims Doctrine. If you don't want to read the post yourself, Bruce essentially posits that, should a player answer a question, there is no way to prove or disprove that the player knew the answer from "fraudulent knowledge" or from "academic knowledge." Indeed, one can come up with any number of alternate reasons why one might have such knowledge, none of which is inherently falsifiable (to give my favorite of Bruce's examples, we cannot show that five-year-old Bruce did not get a cookie while on a bus discussing science with Edward Teller). I'm not a Quizbowl Epistemologist (though it seems that such a position seems called for), so I'll leave it there for people with real philosophy knowledge to expound on.

[setht]

stuff about robots

Okay, this is all very nice to think about, but I think an important point to consider when it comes to human players is that there's a limit to how many "binary-association" type clues a human player can retain at any time. If 80% of the physics questions in quizbowl come from only 20% of the standard physics curriculum and the same thing happens in various non-physics categories, that it makes it possible for a robot-like human to cover a significantly larger swathe of the distribution than if questions spread themselves over the available material in a more reasonable fashion. If a robot-like human figures out that memorizing "Gross-Pitaevskii equation goes with Bose-Einstein condensate" will give it four times as many good buzzes per year as memorizing "analytic functions have harmonic real and imaginary parts" (and probably takes up less brain space), then obviously the smart play for that robot-like human is to memorize the GP stuff, skip learning/memorizing anything about connections between analytic and harmonic functions, and use the saved brain space to memorize other similarly overasked clues. I think it's rather sad that the GP equation has probably been even more valuable for quizbowl (at least in the recent past), relative to stuff like the connection between analytic complex-valued functions and harmonic real-valued functions, than I claimed above--I for one learned quite a bit more about analytic and harmonic functions outside of quizbowl than I did about the GP equation (namely, I saw the former in many classes, and the latter in zero [0] classes). There's no good reason for stuff like GP equation and Autler-Townes effect to be showing up all the damn time while lots of more important material lies neglected--except that it makes it easier for writers to write their questions, possibly.

One possible response to this is to throw up our hands in despair, then settle down to the task of trying to find the best topics to leverage for rote memorization (or quit quizbowl in disgust, or resign ourselves to routinely losing to robot-like humans that don't really understand any of the questions they're answering while they win). A second option is to recall that humans (even robot-like humans) have limited memory capacities and resolve to kill the next several GP equation clues with fire (or, if we're responsible and semi-knowledgeable writers, we could resolve not to use any such clues for a good long time while we turn our attention to various under-asked subjects). If writers/editors make a concerted effort along these lines, the robot-like human is going to take quite the hit on ppg.

-Seth

p.s. I think this thread very much is (or should be) about "how to beat the robots." Dwight has alluded to what I think is the best answer: write questions in such a way that the "robot strategy" is minimized in effectiveness. As I've said above, I think the way to do this is to make conscious efforts to vary answer and clue selection from tournament to tournament. If someone can literally memorize every science clue when the clues range over the whole of the science curriculum (as opposed to memorizing a greatly reduced subset of science clues that just show up over and over again, many of which may not even be in a standard science curriculum), then they're going to get lots of science points off everybody. If they can do the same thing in literature and history (or any other combination of categories that typically gets, say, 13 of the first 20 questions in a packet), they are going to win every tournament they play. I don't think anyone can remember that much material, but the more we as writers go back to the same answers and clues over and over again, artificially reducing the amount of material a robot-like human has to cover, the more viable this strategy becomes.

[grapesmoker]

I will freely confess here and now that I've answered lots of questions based on essentially fraudulent knowledge (see: every music buzz I've ever made, sometimes beating out people who actually listen to music and know stuff about it) or knowledge that was basically picked up from playing quizbowl. This is just how the game is; you remember stuff and you buzz when you know the answer. However, if I were routinely beating music people on music questions without learning anything other than lists of titles (that is to say, without learning something about music or listening to a lot of things), I think we would all agree that this was problematic and maybe we shouldn't be rewarding my ability to associate titles with composers (or at least not reward it more than we would reward some actual knowledge of the music). Likewise, whatever Bruce may say, if Bruce is repeatedly beating me to science tossups (this does not happen in real life), something has gone horribly wrong (or, I guess, Bruce might have decided to teach himself physics, which would be admirable but I don't see it happening), because given the constraints of normal human memory and our respective fields of study, I should be answering something like at least 7 of every 10 science questions vs. Bruce.

Seth is right and this discussion is very much about how to "beat the robot." The whole point is that we don't want to reward the kinds of things that the robot can do (seriously, this is becoming like the Chinese Room argument of quizbowl), and in fact, we know that we can do this. There are some very simple rules we can follow:

1) When possible, describe before you give the name (or instead).
2) Focus on important facts that are likely to be learned by someone studying the subject matter.
3) Explore under-represented topics so that repetition does not allow someone to simply memorize old clues.
4) Try being inventive with your clues by relating them to other branches of study, other applications, etc.

None of this is hard or unreasonable; people write questions like this every day without any barriers. There were lots of questions like this at MO that totally rewarded having studied something or read something, or learned something real about the topic instead of rote memorization. There is no reason whatsoever to think that the current stable of quizbowl writers and editors cannot do this and there doesn't even seem to be any actual disagreement about whether we should be doing something like this. I'm not sure what the actual controversy here is and what purpose such thought experiments serve, but I'm holding, as an empirical matter, that such questions are both eminently possible and desirable.

[grapesmoker]

If someone can literally memorize every science clue when the clues range over the whole of the science curriculum (as opposed to memorizing a greatly reduced subset of science clues that just show up over and over again, many of which may not even be in a standard science curriculum), then they're going to get lots of science points off everybody. If they can do the same thing in literature and history (or any other combination of categories that typically gets, say, 13 of the first 20 questions in a packet), they are going to win every tournament they play. I don't think anyone can remember that much material, but the more we as writers go back to the same answers and clues over and over again, artificially reducing the amount of material a robot-like human has to cover, the more viable this strategy becomes.

If someone can literally perform such a miraculous feat, that person will have won quizbowl forever. Fortunately, since quizbowl is not played by Funes the Memorious, we don't have to worry about this. If someone pulls this off, we can't do anything about it and to tell you the truth, I'm perfectly fine with that.

[cvdwightw]

WARNING: This post is only tangentially relevant to the issues being discussed. If you don't like Dwight Wynne-length posts on quizbowl theory, please skip to the next post in this thread.

New Clues and the Trickle-Down Effect

Consider the introduction of a "new clue:"

At the first tournament, P0 players will buzz off the clue; these are the people with primary knowledge of the clue. The usefulness of a new clue is directly proportional to P0 - if P0 is 0, then the clue is not at all useful. Herein lies the argument against such exciting topics as Fijian myth, which is not likely to be encountered by anyone not doing quizbowl study (this isn't to say that it's impossible, however. I'm reminded of Ray Luo's Indian Rule, which states that "quiz bowl players who grew up in India, or is a child of parents who grew up in India, or who have otherwise some blood ties to India are more likely to get questions on India geography, mythology, religion, history, and literature, regardless of their prowess or amount of experience in quiz bowl...Chinese quiz bowl players are all over the Chinese history, literature, mythology, and classical music...Jewish quiz bowl players are all over the Jewish religion, history, and literature questions, etc, for players of each ethnicity and culture.").

At the second tournament, P0+P1+F1 players will buzz off the clue. This includes all players with primary knowledge, the players who have gained primary knowledge since the first presentation of the clue, and the number of people who remember the clue from the first time it showed up or from studying that packet.

At the third tournament, P0+P1+F1+P2+F2 players will buzz off the clue. Now we are adding players who have gained primary knowledge in between the second and third presentation of the clue, plus those remember the clue from the second time it showed up (F2 being comprised of the people who didn't hear it the first time it came up and remembered off the second time, plus the people who heard it the first time and decided that it was important enough to memorize when it showed up a second time).

We can keep adding more and more players for subsequent tournaments.

This produces an excellent explanation for how fraudulent "old packet" knowledge directly produces the trickle-down effect. It seems to me a rational assumption that the proportion of people hearing a new clue for the first time decays exponentially, until it oscillates between zero and the proportion of people playing their first tournament ever at that tournament (which is reasonably high for novice tournaments, and essentially zero for ACF Nationals). Furthermore, we can consider the probability of recalling the clue after some number of exposures to the clue is roughly sigmoidal (it is difficult to recall the clue after the first few presentations, it becomes easier after some number of presentations, and it is limited by the retrieval failure rate - the knowledge of that clue is there, it just can't be accessed in time, so no amount of increased repetition is going to help.). Obviously the uniqueness and relevance of a clue, and a player's capacity for absorbing lots of fraudulent knowledge at once, is going to change the parameters of the recall equation, but the general shape is the same.

Then we can consider the number of players buzzing fraudulently off a clue to be

Sum over n of (P(number of players hearing the clue for the nth time)*P(probability that a player has recalled a clue after n-1 presentations))

There are (r-1)C(n-1) ways to be hearing the clue for the nth time, where r-1 is the number of times the clue's come up before (this is the rth presentation). Then, the probability that a given player is hearing the clue for the nth time is simply ((r-1)C(n-1))/2^(r-1), and so the number of players hearing the clue for the nth time is expected to be, well, that probability times the number of players. The second probability is a cumulative distribution function - it asks whether or not you've recalled the clue after any of the n trials. This is modeled using, e.g., a scaled and shifted error function (as such, then, the true probability that the player recalls the clue after this particular presentation is simply a scaled/shifted form of the Gaussian distribution).

So the expected number of players that buzz fraudulently off that clue is roughly equal to

Sum from n equals 1 to r of (Number of players*((r-1)C(n-1)/2^(r-1))*sum from k equals 1 to n of (1/2*pi*s)*e^((k-m)/s)), where r corresponds to the rth time the clue's come up, and m and s are the mean and standard deviation necessary to get the correct-looking error function (I suppose that these parameters could be somewhat quantified in an experiment in which a bunch of quizbowl players were given a bunch of nonsense clues and asked to memorize the clue-answer association).

Meanwhile, as the number of presentations increases, the number of players with primary knowledge is likely to increase as well, since some players will presumably be inspired to gain primary knowledge from having heard the clue, or will not have heard (or paid attention to) the clue before gaining that primary knowledge. However, considering that the probability of being inspired by any specific clue is quite low, we can pretty much safely conclude that the driving force behind increased conversion rates is indeed fraudulent knowledge.

The Fraudulent Knowledge Quotient
Because of the Unfalsifiability of Quizbowl Claims Doctrine, we cannot actually measure the Fraudulent Knowledge Quotient, but I'm including the idea behind it here anyway.

The Fraudulent Knowledge Quotient is simply defined as 0 for a 100% conversion rate and (1-conversion rate)*(Number of players buzzing with fraudulent knowledge)/(Number of players buzzing with real knowledge) otherwise, times 100 and rounded to the nearest integer if you want a nice round number. The Fraudulent Knowledge Quotient represents the "fraudability" of the question. New clues will have a very low Fraudulent Knowlege Quotient, since only or mostly people with real knowledge will know them. Giveaways will also have low Fraudulent Knowledge Quotients, since the conversion rate is high and at least some people will have primary knowledge. Clues that are mostly learned through quizbowl knowledge (comedy Autler-Townes reference) will have very high Fraudulent Knowledge Quotients, and stuff that only Ryan Westbrook's robot can get will more likely than not have an infinite Fraudulent Knowledge Quotient.

Clearly, we should be striving to include clues that have as low as possible Fraudulent Knowledge Quotients, while maintaining the proper distribution of conversion rates. This is where middle clues become extremely hard to write, because they will always contain (somewhere in the wide range of middle clues) the maximum in the Fraudulent Knowledge Quotient distribution for that particular questions - these are clues that some people will get, but most people who get them do so off non-primary knowledge.

Reflex Buzzes, Long Term Potentiation, and the Frequency of Clues
Consider the simple idea of a "clue" neuron and an "answer" neuron. The "clue" neuron fires when a clue is recognized, and the "answer" neuron fires when you think you know the answer. There is also a clue-answer synapse in which the firing of the clue neuron causes the firing of the answer neuron.

Meanwhile, another neuron, the "buzz" neuron, is a neural coincidence detector. It fires only when it receives input from both the clue and answer neurons (you recognize a clue, and think you know the answer). The clue-buzz and answer-buzz synapses are quite a bit stronger than the clue-answer synapse.

One might consider a "reflex buzz" to be when the clue neuron activates the answer neuron; thus, both neurons fire and the buzz neuron fires. (There would also likely be some inhibition from other clue-answer associations on this particular system, but we'll ignore inhibition for now, since it's a reflex buzz).

As a clue is used several times, this clue-answer synapse strengthens gradually, and thus there are more reflex buzzes off of "recognize clue, associate with answer." This explains why lots of people get tossups fraudulently off stock clues or giveaways - that clue-answer synapse has been strengthened from months or years of association.

Where it becomes interesting is if there is a (relatively) high-frequency stimulation of a single clue (consider, showing up in almost every single tournament over a period of several tournaments). According to long term potentiation, high-frequency stimulation will quickly strengthen the synapse, and its strength will persist after a long time of inactivity. Using the clue-answer-buzz paradigm, then, we would expect that this high-frequency stimulation would strengthen the clue-answer synapse, and this synapse would weaken quite slowly.

Thus, we might expect that even if people decide to stop writing on Danticat for one or two high-level tournaments, this might not be enough time for that Krik? Krak! - Danticat clue-answer synapse to go back to "normal" levels. If this analogy between long-term potentiation and quizbowl learning is at all on the right track, then to have the best chance at stopping fraudulent knowledge, we not only need to include new clues but also (as Seth alludes to) make a conscientious effort to eliminate old clues and answers whose Fraudulent Knowledge Quotient is high. Thus, we can allow those clue-answer synapses to slowly decay until that clue-answer association has become extracanonical and people have somewhat forgotten about it.

[No Rules Westbrook]

Bruce's Falsification Theorem is pretty brilliant, and it's one good reason why we shouldn't concern ourselves with punishing or rewarding the robot.

[Sargon]

There are several ways to stop the robot other than perpetual canon expansion, which I am convinced will slowly strangle quizbowl to death in a sea of preposterously obscure answers that perhaps a dozen people on earth find enjoyable to play on. I am not advocating either per se, but I think they would solve the problem and might be able to be made into something workable.

First is to start enforcing eligibility requirements on many tournaments. There is a limit to what you can learn in 2-4 years so Junior bird or undergrad only tournaments could conceivably be kept at a reasonably constant difficulty (and in principle 2 year grad students, 4 year grad students, etc. could be made as well). By the time the robot has gotten a strong fraud knowledge base it will cease to be able to play. One could also gear most tournaments for the less experienced demographic and have older players simply accept that they are good and will be buzzing a lot on the first line with fraud knwoledge.

The second is to carefully the reduce the amount almost any given thing comes up both as a clue and an answer to at most a few times a year. In principle the robot learns after the first occurrence, but in practice most people learn only after an answer or closely related answers have come up several times. When that year's one element of the set of the third most famous Gambian author, his family, and his work comes up, the one quizbowler who has read something of his will answer it and hardly anyone else will remember it, ensuring that player a good buzz on it based on primary knowledge in the future. Exceptions would have to be made for things like Dickens and Shakespeare (perhaps limiting each book or play to once a year), but it seems to me there are enough answers out there to do it. Moreover, this would put an end to things being "quizbowl famous." Doing this would perhaps require the creation of a used answer registry for the year, perhaps to be updated immediately after each tournament. While one could read the list or compile and read one and infer that as Mansa Musa's Vizier's cousin has come up twice, he is done for the year, the size of all potential remaining answer would make this edge negligible.

[setht]

Bruce's Falsification Theorem is pretty brilliant, and it's one good reason why we shouldn't concern ourselves with punishing or rewarding the robot.

I disagree with everything you just said, Ryan (surprise!).

I think Bruce's Falsification Theorem, like most attempts at quizbowl philosophizing, has almost nothing of practical interest to offer writers, editors, or players who are trying to write, edit, or play questions. In the particular case of this thread, the fact that we are unable to point at any single buzz and conclusively determine whether that buzz was fraudulent or not seems irrelevant to me: if we want quizbowl to be a game that privileges real knowledge over fraudulent knowledge as much as possible, then writers and editors should take steps to make sure that widespread methods of gaining fraudulent knowledge are not systematically rewarded. It seems clear that playing lots of quizbowl/studying old packets is a widespread method of gaining fraudulent knowledge, and it seems clear that an appropriate response to that particular method of gaining fraudulent knowledge is to make a conscious effort to spread our selection of clues and answers among as large a pool of worthwhile clues and answers as possible.

If everyone were to select answers and clues randomly from the full pool of worthwhile material, there would be statistical fluctuations, but no robot could predict which things were due to come up more and would have to study the entirety of the pool--a task I have argued is beyond anyone's capabilities. The problem occurs when people are attracted to past answers and clues. Given that there's some amount of "passive fraudulent learning" among all players, such that high-level players who aren't particularly interested in learning all about Danticat can still probably name multiple Danticat works at this point, I think it might be good to go beyond the dictum "choose clues and answers randomly" and ask writers and editors to make some conscious effort to avoid topics that have come up a lot in the recent past.

-Seth

[mujason]

The robot is coming, at least to Jeopardy!

http://www.nytimes.com/2009/04/27/techn ... .html?_r=1

[grapesmoker]

The robot is coming, at least to Jeopardy!

http://www.nytimes.com/2009/04/27/techn ... .html?_r=1

Any chump can play Jeopardy. We need to notify them of where the real challenge lies.

[theMoMA]

I actually am kind of interested in Bose-Einstein condensates, so I could tell you that the Gross-Pitaevskii equation is often conceptualized as a nonlinear version of Schroedinger's equation, and also that it's somewhat analogous to the Ginzburg-Landau equation. I don't actually know where or why I learned those things, either; those are simply the things I know about the G-P equation. I'm certainly not a physicist, and I don't purport to know much about the mathematics of physics beyond the basic Newtonian mechanics, optics, and electronics that I took in high school. But I could talk to a physicist about BECs and not feel out of place.

The point isn't to say that my physics knowledge is great, because for one thing, it's not. I'm just saying that there are a lot of things that we learn to many varying degrees by osmosis or by writing questions or simply out of curiosity. I think this thread is getting carried away with calling various types of knowledge less desirable or less valid than others; it seems to me that this knowledge is simply less complete. Like I said, I feel like I could have an intelligent conversation with a physicist about BECs, but I'd be the one asking all the questions, trying to figure out how to conceptualize operators, trying to make up for the fact that my lack of education in the system of science has left me at an extreme disadvantage of understanding. Similarly, if someone who knows that "On his blindness" is a poem about becoming blind that was written by John Milton talks to noted person who has studied that poem in a class me, they will also be at a disadvantage of understanding.

Quizbowl questions should obviously reward the person with the best, most complete understanding of the topic at hand. Framed in those terms, it's easy to see why people who learned about BECs in class should be getting tossups over people who know that G-P = BEC. It has nothing to do with some idea that learning in class is inherently better than other types of learning, or that G-P is not emphasized in the typical curriculum. It has everything to do with the fact that the person who learned about BECs in class has a much more complete understanding of them. If some pioneer in BEC science from Lawrence-Livermore, or some self-taught BEC savant with tons of legit knowledge shows up to the tournament, they should hopefully get a BEC tossup before people who simply have some class experience with them, etc.

You can apply the "complete understanding" doctrine to figure out why any form of learning something should or shouldn't be privileged over any other; obviously someone who has read a book understands it better than someone who memorized every character name. Someone who spent a summer researching the March to the Sea will understand it better than someone who knows the name of every minor Civil War skirmish.

[No Rules Westbrook]

Outside of the Falsification Theory, my proposal is that the robot is always going to win. There's nothing you can do, so there's no sense in trying. If you think you can beat the robot - you either overestimate the power of "real" knowledge and understanding in this game, or you understimate the power of the robot.

Getting preoccupied with the robot or the "idea of the robot" won't solve anything - and it may well lead to poorer questions on the whole. If you just ask whether every clue you give is important and rewarding of knowledge, you'll do fine - if you start combobulating your questions to defeat the robot, all sorts of bad things are going to happen.

[Captain Sinico]

...my proposal is that the robot is always going to win... If you think you can beat the robot - you either overestimate the power of "real" knowledge and understanding in this game, or you understimate the power of the robot.

This is begging the question. The question in your terms must to be: "Can quizbowl be changed so that real knowledge is rewarded more strongly, perhaps to a point that a robot with total recall of every previous clue can be beaten by someone with a high degree of real knowledge?" Statements about what quizbowl is now are utter failures to engage with the discussion, which is about what quizbowl ought to be and, in fact, should be counted as denunciations of what the game is now.

Even with this modification of your statement, your analysis is shockingly deficient in key areas. In the first place, you have not set up a fair adversary (or, really, any adversary) for your robot. Certainly, in any fair version of quizbowl, a robot that has learned and has total recall of the contents of, say, every textbook published ought to do at least as well as your robot, which knows only quizbowl clues. Relatedly, you have made no assumption about non-robot players' recall of non-new clues, while simultaneously arguing that any good player must have a command of these. In the third place, even a more complete (or, at least, internally consistent) neighbor of this argument necessarily neglects a number of important facts, among them the fact it's probably possible to write a entire tournament of new* answers and clues that would otherwise be okay by whatever definition. Finally, even if you account for all of these objections, it is highly questionable what utility your analysis has, given that no robot-as-defined-by-you-like entity is likely to exist as a quizbowl player and an analogous entity with even one of the assumptions of complete perfect learning of every new clue forever; concomitant infinite storage space; and total, instantaneous recall of everything learned relaxed becomes eminently beatable without any major changes to the game. In particular, a more realistic model of learning, like the one Dwight has posited, adduces strongly the point that the overuse of clues, especially for things about which little real knowledge or understanding is likely to be available to/commanded by players, is massively, unduely rewarding to the formation of shallow associations from old rounds, necessarily to the exclusion of other, more worthy learning pursuits.

In fact, Ryan, if your argument obtains and has any relevance, why is it wrong to re-use questions entirely? After all, the people who've memorized the old rounds are just going to get the questions anyway, right? But, of course, this is absurd; such a game is not fun and doesn't encourage learning by any good definition of the word.

Incidentally, the idea that we can obtain no measurement of how likely it was that someone's buzz was fraudulent (by some definition) is intellectually bankrupt, call it by whatever insipid capitalized name you like. On the contrary, the data required to make such a judgment (what things people seriously study and how, to start with) seem eminently available to a fairly high degree of precision. Even were that not the case, it requires an unbelievable level of stupidity to be unable to imagine a situation in which such data would not be available.

Finally, to respond to Paul's concerns, I don't think anyone thinks a solution here is one of perpetual canon expansion of the type he's rightly afraid of; it's certainly not mine. On the contrary, my problem with the game is that the canon has, in fact, contracted in some ways and clues and answers have concomitantly systematically concentrated at a number of points of questionable merit. In particular, these concentration points are often discordant with what someone designing a game from a perspective of serious academic study would emphasize. It is my conjecture that this is happening and has happened due in large part to the emulative nature of writing as advocated by many and a reliance on conversion rates as the measure ne plus ultra of question fitness, but I suppose it's really not important why such an issue exists at the moment, since we seemingly can't agree on what the issue is.

MaS

*This is conjecture, but not one that I think can be dismissed on its face. I think this proposal gets increasingly plausible if we consider anything that hasn't come up in several years to be new. I know Ryan agrees with this to some extent, in fact, given his discussions of a "lost canon."

[The King's Flight to the Scots]

You can tell people to get "real" knowledge as much as you want, but until there's a definite method of improving other than by reading old questions, people will keep amassing "fraudulent" knowledge that way. However, it seems to me that studying old packets helps me learn quizbowl relevant information, and otherwise, I have no idea where to start. Just look at all the topics about "ways to get better": Nearly all of them say "study packets." It seems disingenuous to tell people to improve that way, then say that it's a false method of learning. A list of resources that could provide a definite method of improvement while simultaneously teaching real knowledge would help this situation immensely. Just consider how goddamn difficult ACF Nationals is. If you're going to encourage even high school teams to play at that level (and you do), you need to provide a better way to improve than "study this subject exclusively for 8 years".

I'm not deadset on anything in this post: if anything I said is untenable, feel free to show me why I'm wrong. However, I do think it's necessary to ground ourselves in concrete examples instead of abstract theory.

[grapesmoker]

Just to be clear about ACF Nationals: this isn't a tournament that caters to high schoolers or relatively weak college teams. If you come to ACF Nationals, be prepared to play a hard tournament. Saying, "learn a lot of stuff to do well here," is entirely appropriate at the premiere national quizbowl championship.

[Mechanical Beasts]

Yeah; if you are a high schooler, then (unless you do research with boron nitride nanotubes, giving you tons of legit giant Stark effect knowledge) your "real knowledge" will probably come from literature and history and, like, things that you can read. (And, of course, some of the science; I exaggerate.) ACF Nationals won't work for you unless you have some fake knowledge or unless you're truly remarkable, and that's how it should be.

[Captain Sinico]

Matt, that's also begging the question. Your argument is: "One ought not change quizbowl to reward superficial word associations from old packets less because quizbowl rewards those things at a given level now and/or people know it does and told others so." Why, then, was it right to deemphasize, say, the list of Nobel laureates, or the multiplication table, or literally anything else that anyone found useful ever? I think that claim is sufficiently ridiculous to collapse on its own without further note from me.

On a similar note, I'd like to call to the attention everyone (else) who's thinking of making such an argument to the facts of the mater. To wit, we're playing and discussing a game that has changed to a considerable extent due to the explicit design and influence of a small number of people on a fairly short time scale. Especially note that the types of things the game asks about and how it asks about them have changed radically. Support of the notion that the game can't* change from its current state in any way consequently requires a degree of myopia that I am appalled to find in any thinking observer of the game. The claim that the game can't be made to or ought not to or ought not be designed to emphasize certain types of knowledge and understanding over others is equally sophomoric. Quizbowl already does emphasize and always has emphasized certain types of knowledge and understanding over others, and that largely by explicit design. In the past, the same thing was the case for different kinds of knowledge and understanding. So I hope that any further such manifestly wrong arguments will fail to reach and annoy me and others, hopefully due to having died at the well I just poisoned the living shit out of.

MaS

*If, conversely, someone wants to claim that quizbowl ought not change from its current state, that's a tenable claim, but one with which I am not alone in disagreeing.

[The King's Flight to the Scots]

Matt, that's also begging the question. Your argument is: "One ought not change quizbowl to reward superficial word associations from old packets less because quizbowl rewards those things at a given level now and/or people know it does and told others so." Why, then, was it right to deemphasize, say, the list of Nobel laureates, or the multiplication table, or literally anything else that anyone found useful ever? I think that claim is sufficiently ridiculous to collapse on its own without further note from me.

Well, I'll clarify that a bit: my argument isn't that quizbowl should reward superficial word associations. My argument is that if you want to change the way the game is played and written, you should give people concrete sources from which to do so. Right now, it seems (to an inexperienced and on the whole, mediocre player) that the way to improve at the current game is to read old packets. This isn't an optimal situation; however, at least it's a method by which a novice can definitely improve. The science topic a while back was great at providing real sources and textbooks to learn from; I think it would be great if people could provide analogous methods to acquire "real knowledge" in other subjects. Consider, also, that many of the questions for ACF Regionals are not written by expert players. As posters in this topic have stated, those writers are mostly emulating what has already been written. To change that, I think it is necessary to provide concrete, academic sources to take clues from.

On another note: thanks for responding.

[marnold]

Man, this thread sorta sucks. No one is saying to stop reading old packets to improve, just that the game shouldn't reward that more than it rewards knowing stuff from a more legitimate avenue. I find it hard to believe that people don't have an innate sense about how to learn stuff besides by memorizing old packets, or find clues besides looking in old packets. Maybe the most concise advice is this.

[Captain Sinico]

Then what I'd say is that, in the game I'd like to play, you'd be able to improve best by undertaking serious study of whatever topics. So, read a book (pre-empted to this point by Michael Arnold/my boy D-Mite), check out some course notes, attend some lectures, take classes, listen to or take lessons in music, look at paintings, etc. Obviously, I can't tell you how best to improve at the actual future version of the game because I can't say for sure what it will be. I also don't think there's a single best way to improve, even for a fixed game, because people learn differently enough, but those are secondary concerns. What is primary is that I can't imagine any form of quizbowl for which some effective means of improvement wouldn't exist.
I also don't think that reading old rounds would be a bad idea, even in my ideal form of the game, which would reward deep academic knowledge and understanding as I see them much more highly than the current version of the game, especially relative to word associations from old rounds. I continue to stress that the type of superficial knowledge obtainable from old rounds should be and probably must be rewarded at some level; I just wish the level were lower than it is now.
To take up Andrew Hart's example, which I think is instructive, if one knows that the Gross–Pitaevskii equation gives the ground-state single-particle wave function of a Bose-Einstein condensate and/or is a type of Schroedinger equation with a additional potential term proportional to the square modulus of the ground-state single-particle wave function and/or maybe even that it's "often conceptualized as a nonlinear version of Schroedinger's equation" (which I would argue is tellingly vague, but that's neither here nor there,) then one should be able to do better than someone who merely knows its name, for example.
My fundamental point, then, is that there's a much richer world of learning than the mere words there. One can't even really know what most of those things mean from just reading old packets. I speak as someone who's taught upper-division quantum mechanics several times, which experience lets me confidently say that even bright students well into science education at a university level often struggle with the meanings of many of those subtle concepts. So, a better question ought to do a better job of tapping into and discriminating on the basis of depth of penetration of that world of hard-earned academic understanding of what this equations is and what it means and only then reward more superficial knowledge, like what can be gleaned from reading old packets. So, even if I were the sole dictator of how quizbowl is written, it wouldn't necessarily be a bad strategy to read old packets to obtain some breadth, for example; I would think that, even in the type of game I have in mind, that would be a strategy comprising at least part of the optimal mix of training for many players.

MaS

PS: "Yo' [game] needs [knowledge-and-understanding-rewarding questions] / So [write] that shit!"

[magin]

To make a point that might help writers and editors, I think there's a continuum running from clues with high context, clues with low context, and clues with almost no context. To elaborate:

The clue "the GP equation is associated with these things" doesn't seem like it has much, if any, useful context that will differentiate people who understand science and people who memorize words. I'd like to see these kinds of clues significantly decreased, since I don't think they serve a purpose that more contextual clues can't also fill.

Clues with low context, on the other hand, are more rewarding to people with understanding. For instance (to use an example from literature): The clue "This author described Mugo's betrayal of Kihika in A Grain of Wheat" has more context that rewards understanding than mere word associations, but is low-context because it's also prone to being memorized. In my opinion, such "low-context" clues are fine for the second half of a tossup, but more problematic for the leadin or early middle clues because of their susceptibility to context-free buzzes.

Clues with high context are clear to people who understand the topic and know the clue, but are hard to memorize. For instance, Jerry's example "These phenomena are described by a nonlinear version of the Schrodinger equation with an operator that includes a term proportional to the square of the wavefunction magnitude" seem to fall in these categories. I would prefer for these clues to always precede low-context clues in tossups, and would like to see more of them in general, because questions that reward rich understanding can consist mostly of high-context clues, whereas questions consisting mostly of low-context clues are much poorer at differentiating levels of understanding (and questions consisting mostly of very low-context clues are extremely poor at it).

EDIT: When I say "hard to memorize," I don't advise people to obfuscate their clues; good clues are clear, not frustratingly vague or coy. I mean clues that are both clear and resistant to rewarding one-to-one associations with an answer.

ANOTHER EDIT: I'm also fine with using low context clues at the beginning and early middle of tossups if those clues haven't been used much, or at all, before, to clear that up.

[grapesmoker]

Hey everyone: stop writing science questions that contain clues of the following form: "Phenomenon X can be modeled by named equation Y," unless you have verified that Phenomenon X is nearly-exclusively modeled with Equation Y. For example, Lederberg contained a clue for Ising models that said something like "These can be solved using the Metropolis algorithm," which is almost certainly true for a wide range of things. Another similar example was found in THUNDER (obviously the answer and the clue were different but the principle was the same). These kinds of clues are not usually helpful, especially when they involve common analytic techniques that can be applied to a large number of phenomena.

[grapesmoker]

Sorry for sort-of resurrecting this thread, but...

I was uploading Chicago Open 2009 to the archives this afternoon, and since it was improperly formatted, I had to reformat the set. Which meant that I ended up skimming various portions of it and I found something truly disturbing, which is the extent to which the material from that set has percolated into other later sets scant months after its occurrence. Quizbowl as a whole really needs to get better about this; you can't just return to old packets and especially not to tournaments like CO for your answer choices, and yet a shockingly large amount of tournaments appear to operate on this model.

[Cheynem]

This has nothing really to do with mainstream quizbowl, but I will also say that bar none one of the worst offenders of clues that don't really reward understanding is trash packets. People assume that I watch a lot of TV because I can answer television questions. That's not really true--rather, I read a lot about TV like in TV encyclopedias. This allows you to answer questions because so many lazy trash questions go "Cast members of this show include _____" or other easily memorized stuff. I just answered a question about "Judd for the Defense" at practice. Have I ever seen an episode of this? No. Do I know what it is about? Not really. I know it starred Carl Betz though because that's one of the things I read about in a TV encyclopedia. Now this is a fairly ludicrous example, but my basic point is that trash in my opinion tends to reward robot knowledge even more than regular quizbowl.

[No Rules Westbrook]

Reading about something in a tv encyclopedia is completely legit knowledge - it's not firsthand, but a big part of this game is how much we know about stuff we've never experienced "firsthand," and thats how it should be.

Now, sure - the question should be pyramidal - and someone who's watched every episode of "Judd for the Defense" and has the DVD set at home (cause, hell, there probably is one) - should obviously be able to get it first.