ACF Nationals 2015 Discussion

Old college threads.
User avatar
Birdofredum Sawin
Rikku
Posts: 400
Joined: Tue Nov 09, 2004 11:25 pm
Location: Mountain View

Re: ACF Nationals 2015 Discussion

Post by Birdofredum Sawin » Tue Apr 21, 2015 12:13 pm

No Rules Westbrook wrote: But, it does agitate me a little when people keep citing - as a defense for that - the same 6 or 7 tossups. Unlike most editors today, I will throw in a small handful of tossups that are just way outside the usual difficulty. I'll "take a few shots," as I term it - and you don't really see many editors do that these days. But, it's literally like 5-6 shots in the whole tournament.
One of the great charms of Westbrookianism is that every time you think you have reached the lowest deep of its wackiness, a lower deep threatening to devour quizbowl opens wide. Here is a prime example: not just acknowledging that you sometimes write tossups that you don't expect anyone to answer (which is what "just way outside the usual difficulty" has to mean, in pragmatic terms), but taking a bizarre macho pride in doing so!

To be clear, the approach described above is really, really silly. If this thread has taught us anything, it should be that there is significant variance between editors' abstract, on-paper notions of "what is gettable" and what turns out actually to be gotten by teams. Maybe your assumption that "everyone knows David Dacko" was way off-base. Maybe you chose your clues poorly and an otherwise-gettable tossup goes dead because people can't get to the answer from the clues you provided. Maybe, in the course of a punishing day of quizbowl, the only person in the room who knew the answer took a dumb neg. The point being, what seems "gettable" when you are sitting calmly at home reviewing your materials may be far afield from the "lived experience" of the questions when they are actually played.

Experienced editors are, or should be, well aware of this phenomenon. That's why such editors err on the side of conservatism--they know that questions invariably play harder in real life than they seem on paper, and that invariably some of the questions they thought were "middle-of-the-road" will turn out to be quite hard, while some questions they thought were "quite hard" will turn out to be impossible.

Getting back to the absurd Westbrookian formulation that triggered this post: Editors don't need to "take a few shots" (in the Westbrookian sense), because the set will already do that for them. That is, it's already going to be the case that things the editors thought were well within the bounds of gettability will happen to elude the field; it's ridiculous to add on top of that yet more questions that you expect the field not to get.
Andrew

Ex-Virginia, Ex-Chicago, Ex-Stanford

Adventure Temple Trail
Auron
Posts: 2613
Joined: Tue Jul 15, 2008 9:52 pm

Re: ACF Nationals 2015 Discussion

Post by Adventure Temple Trail » Tue Apr 21, 2015 12:29 pm

Birdofredum Sawin wrote: Here is a prime example: not just acknowledging that you sometimes write tossups that you don't expect anyone to answer (which is what "just way outside the usual difficulty" has to mean, in pragmatic terms), but taking a bizarre macho pride in doing so!
Yaphe is absolutely right about this. If you are writing a ten-line tossup just to see if teams could get it somewhere near the end or not, that tossup reflects a failure to understand what tossups are even for or what they are designed to do. That same content could be retrofitted to a more gettable answer, compressed to a few clues into a larger tossup on another answer, saved for a bonus part, or just recorded as a short lecture and uploaded to the Internet as an educational podcast if you really think it's something the quizbowl community wants to know about (but is too afraid to ask).
To be clear, the approach described above is really, really silly. If this thread has taught us anything, it should be that there is significant variance between editors' abstract, on-paper notions of "what is gettable" and what turns out actually to be gotten by teams. Maybe your assumption that "everyone knows David Dacko" was way off-base. Maybe you chose your clues poorly and an otherwise-gettable tossup goes dead because people can't get to the answer from the clues you provided. Maybe, in the course of a punishing day of quizbowl, the only person in the room who knew the answer took a dumb neg. The point being, what seems "gettable" when you are sitting calmly at home reviewing your materials may be far afield from the "lived experience" of the questions when they are actually played.

Experienced editors are, or should be, well aware of this phenomenon. That's why such editors err on the side of conservatism--they know that questions invariably play harder in real life than they seem on paper, and that invariably some of the questions they thought were "middle-of-the-road" will turn out to be quite hard, while some questions they thought were "quite hard" will turn out to be impossible.

Getting back to the absurd Westbrookian formulation that triggered this post: Editors don't need to "take a few shots" (in the Westbrookian sense), because the set will already do that for them. That is, it's already going to be the case that things the editors thought were well within the bounds of gettability will happen to elude the field; it's ridiculous to add on top of that yet more questions that you expect the field not to get.
I differ with Yaphe here in that I think it is earnestly fine for a set to have a few "deliberate outliers" to keep teams on their toes and/or get some stuff in there to help separate out the best of the best (i.e. top bracket teams at the events it's played at, not the 1st and 2nd place teams). I remember being very unnerved by Nats '14 taking this approach, and then coming around to it in set discussion (indeed, so much so that I wrote several for the ICT that Yaphe just edited, and allowed a tamer analogue of same in the Regionals I edited.) I imagine Westbrook agrees with me as a matter of principle, as he's said previously in this thread a few times. My issue with this set is not that it chose to push the envelope a few times per packet, but rather that an inordinate number of "oh, this tossup on [hard thing] will play fine; it's come up a lot" judgments were either overly optimistic or wildly off-base, resulting in lots more ExxxTREME tossups and middle parts than the past three Nats sets. Seems like this happened far less in Billy and Ike's output than in Rob's and Ryan's. Obviously a lot of teams were still able to roll with it, and by this point Ryan has already conceded that future Nats events will probably ease off the gas pedal a bit in this particular way, so that's all well and good.
Matt J.
ex-Georgetown Day HS, ex-Yale
member emeritus, ACF

Sailing away on my copper boat

User avatar
Birdofredum Sawin
Rikku
Posts: 400
Joined: Tue Nov 09, 2004 11:25 pm
Location: Mountain View

Re: ACF Nationals 2015 Discussion

Post by Birdofredum Sawin » Tue Apr 21, 2015 12:54 pm

I'm not sure that Matt and I disagree, though I am fairly certain that Ryan and I disagree.

To be clear, I'm not arguing that every question in a set--especially a high-level set like ICT or ACF nats--has to be on something highly canonical and definitely gettable. Invariably, there is going to be a mix of tossups on "easy answers but with hard clues" (e.g. an ACF nats-level tossup on something like "Tolstoy") and "hard answers, period" (e.g. a tossup on "the 14th most important poem in Wallace Stevens's first book of poetry"). And invariably, editors at such tournaments are going to want the mix of "hard answers" to include new and challenging material. I think that's all fine and good.

I think the issue, again, may come back to the conventional (but not, I think, inaccurate) perception that someone like Ryan derives his sense of what might constitute "new and challenging" material by scraping old packets for lead-in clues and hard bonus parts, while other people come by it in a more "real world" way. For instance, I remember people saying that a Colson Whitehead bonus I wrote for last year's ICT was "incredibly hard." I understood when writing the question that Whitehead might be "new and challenging," but I asked about him because Whitehead is a significant contemporary novelist whose works are routinely reviewed in publications like the New Yorker, the NYT, etc. So, in asking about him, I wasn't thinking "this is way outside the usual difficulty; time to knock some lit players on their heels" (or whatever), but instead "this may not be something quizbowl asks about yet, but it is a significant topic that people with an interest in contemporary literature can be reasonably expected to be familiar with."

Again, I suspect Matt would agree with this perspective (certainly my perception of his "new and challenging" questions in this past year's ICT was that he was operating on something like this approach). I have no idea whether Ryan would agree with it, but that's unsurprising, given that I find his approach to the game unfathomable in so many ways.
Andrew

Ex-Virginia, Ex-Chicago, Ex-Stanford

User avatar
Tees-Exe Line
Tidus
Posts: 622
Joined: Mon Apr 12, 2010 5:02 pm

Re: ACF Nationals 2015 Discussion

Post by Tees-Exe Line » Tue Apr 21, 2015 12:55 pm

I've been asked for a more complete philosophical statement, so I shall oblige.

I am absolutely certain that Ryan is one of the most, if not the most, hard-working editors in quizbowl. And I sympathize with having one's hard work raked over the coals for sometimes-petty, sometimes-inconsistent, sometimes-self-interested, always-hypocritical reasons. He is also, in a way, one of quizbowl's most innovative contributors. But not in a way I think is healthy, and this is critical enough to the core purpose of what we're all doing here that it's worth a frank discussion.

I have one basic objection to the Westbrook philosophy, but that objection embraces both its implications for extrinsic worth and for quizbowl play. It is this: piling obscurata atop obscurata is just. not. history. The whole revolution of modern historical scholarship is that you just aren't correctly recording and interpreting the past if you list facts, especially de-contextualized ones. It isn't knowledge, even if, in some bare bones sense, it is "true." It would be like writing an earth science tossup that takes Neptunism as its starting point.

And writing quizbowl questions that way does not, in fact, reward serious studying at the expense of a passing acquaintance with the subject. It rewards soul-destroyingly-obscure list-based studying at the expense, as I said above, of reading and comprehending good books. I don't think you need to be a professional historian (which I am not) or a fanboy of professionals (which I am) to learn proper history, and one doesn't need to follow their lead regarding subject matter to have internalized the basic viewpoint I'm expressing. For instance, it can be at the heart of a treatment of military history or of "high" political history, even though those things are out of fashion among professionals. Here is a book that I love, partly because it's written by my dear friend (and ill-advised Urgent Call for Unity answer line) Bernard Weisberger, partly because it's just incredibly illuminating about the events at its core and their larger context. It is a history of the 1906 World Series. I read it right as I moved to Chicago and it provided me great insight into Chicago's history and the resulting legacy that is, I would assert, critical to understanding the city now. It is a book about Baseball History.

Getting back to this particular tournament, there were tossups with gettable answer lines, tossups with answer lines that turned out not to be gettable but would be in an ideal world (Old Court/New Court, Billy Sunday, Maji Maji War), and tossups with answer lines that should never appear (Selous Scouts, William Pendleton). But what united all of these was the overwhelming dominance of obscure, uninsightful, decontextualized clues that, if they were to inform people's engagement with history, would make it about the worst intellectual endeavor I can imagine.
Marshall I. Steinbaum

Oxford University (2002-2005)
University of Chicago (2008-2014)

Get in the elevator.

User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4026
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: ACF Nationals 2015 Discussion

Post by Auks Ran Ova » Tue Apr 21, 2015 1:15 pm

Matthew J wrote:"oh, this tossup on [hard thing] will play fine; it's come up a lot"
While I would dispute this characterization of my editing philosophy, I agree that my categories (mostly lit, I think; people seemed to enjoy music and myth with fewer issues), especially in the editors' packets, tended to skew a little too heavily towards the outlier areas. I've been staying pretty quiet in the discussion thus far because I think having a good sense of people's reactions, unmuddied by my arguing with them, will help me adjust my approach for next year.

I do want to extend special thanks to all the submitting teams this year for a by-and-large excellent bunch of raw questions in my categories. Suffice it to say I had to do a lot of work for this tournament on a fairly restricted time schedule, and having already-extremely-usable questions to work with helped a ton. A lot of the category overloads (Poland, Beethoven) came as a result of my deciding to use good questions where I had them rather than spend costly time micromanaging. Next year I shouldn't have to make that trade, but this year thanks to your great questions it was one I didn't have to feel too bad about.

EDIT:
Birdofredum Sawin wrote:So, in asking about him, I wasn't thinking "this is way outside the usual difficulty; time to knock some lit players on their heels" (or whatever), but instead "this may not be something quizbowl asks about yet, but it is a significant topic that people with an interest in contemporary literature can be reasonably expected to be familiar with."
I think this accurately represents the relevant aspects of my lit-writing philosophy as well, and that thoughts along these lines are why you got things like the Dictionary of the Khazars tossup or the bonus part on Patricia Lockwood. Obviously you're never guaranteed a 1:1 correspondence between what an editor thinks the field will know and what the field will actually know, but that's always at least the goal.
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT

User avatar
Ike
Yuna
Posts: 917
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by Ike » Tue Apr 21, 2015 2:22 pm

As someone who worked and chatted with Ryan on this tournament - I don't think he ever thought "Man, I'm going to blow someone's mind" by frustrating them with an answer that has a 0-10% chance of being answered by one person - what it earnestly seemed like to me was that "I'm prepared for the possibility that no one will answer this." I think the only time he said "oh man, people's heads are going to explode" was when he stumbled across my 5-5-10-10 bonus because of it's "old-timey" feel.

I'll take this moment to note that I think Ryan's approach is actually psychologically more tame if it were to be implemented it at the ICT. I wrote a number of questions for ICT with the mindset of "very tough" - though I heard they were all converted, and some of them were even first clued! In ICT it's a four-line tossup, and if you don't know it you just moved on to the next question right away. Since many moderators can't get through all 24 questions, you hear the same number of tossups anyway, since that only sucked out 20-30 seconds of reading. In ACF, after 8 lines of material, there's that awkward five second silence in which you feel the earth sink as you try to come up with the answer and then the awkward feeling of not getting the answer. Furthermore, many players at ACF treat each individual question on a particular topic at Nats as "their category" and it hurts when they can't get it; at ICT you really don't have the time to think about each question and dwell on the fact that you suck because you can't get a certain question.
Ike
UIUC 13

User avatar
Nabonidus
Wakka
Posts: 113
Joined: Sat Apr 19, 2014 9:32 pm

Re: ACF Nationals 2015 Discussion

Post by Nabonidus » Tue Apr 21, 2015 3:23 pm

Well, I agree with Marshall. I'm not a history player, but the notion of "list-based studying" sounds pretty darn odious to me.
Derek So
McGill

User avatar
Muriel Axon
Tidus
Posts: 698
Joined: Wed Mar 21, 2012 12:19 am

Re: ACF Nationals 2015 Discussion

Post by Muriel Axon » Tue Apr 21, 2015 4:17 pm

Nabonidus wrote:Well, I agree with Marshall. I'm not a history player, but the notion of "list-based studying" sounds pretty darn odious to me.
I'm sympathetic to the Steinbaum-ian perspective here, as anyone who knows me would know, but I don't think this is a fair characterization. I'm sure that if someone got a good buzz on a Westbrook-style tossup (like the Old Court-New Court controversy), it's still more likely that they have substantial knowledge of what it was about aside from just "list-based studying."
Shan Kothari

Plymouth High School '10
Michigan State University '14
University of Minnesota '20

User avatar
Nabonidus
Wakka
Posts: 113
Joined: Sat Apr 19, 2014 9:32 pm

Re: ACF Nationals 2015 Discussion

Post by Nabonidus » Tue Apr 21, 2015 4:36 pm

Oh, I don't disagree from an in-game perspective. By "odious" I merely mean "boring, time-consuming and probably not useful outside of quiz bowl".
Derek So
McGill

User avatar
No Rules Westbrook
Auron
Posts: 1222
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by No Rules Westbrook » Tue Apr 21, 2015 6:06 pm

Well, here's what I was trying to argue with my post.

If you want to look at a packet where 8 questions went dead in a round in the third bracket and say "that's too many" - then I'm pretty sympathetic to your argument. I was reading in the top bracket, but I do know what it's like in the other brackets, and I'm sensitive to that - especially for a team that's really trying its butt off, but too many questions are going dead to have a competitive match.

But, instead of constantly focusing on that one balls-hard tossup on the Selous Scouts, let's focus on the 7 other questions that went dead in that packet and see what the deal is with those. As Ike said very well, my attitude when writing those small handful of tossups on stuff like Billy Sunday was "I'm fine if this doesn't get answered...it's one of the handful of hardest tus in the event, and if it goes dead, it goes dead." (not anything like "take this, suckers!" or "watch how crazy this is!")

Like, really, go and look at my list of history tossup answer choices - and let's talk about the brunt of those. Not two or three outliers. A lot of my arguing in this thread is because, on the whole, I find the vast majority of those answer choices to be quite tame and responsible (at the very least in the sense of being solidly below what I'd consider "open-level" difficulty.) Everyone knows I have wild tendencies at open-level events, but I am awfully cognizant of being responsible with answerlines whenever I write for National tournaments (which I have for several years now).
Last edited by No Rules Westbrook on Tue Apr 21, 2015 6:30 pm, edited 1 time in total.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
Mewto55555
Tidus
Posts: 708
Joined: Sat Mar 13, 2010 9:27 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by Mewto55555 » Tue Apr 21, 2015 6:20 pm

Playoff tossups answered per game, presented without significant elaboration:

(all stats from http://www.hsquizbowl.org/db/tournament ... /playoffs/ and http://www.hsquizbowl.org/db/tournament ... /playoffs/)

2014, top bracket (1-12): 17.15 (1132 through 66 games)
2014, second bracket (13-24): 15.77 (1041 through 66 games)

2015, top (1-12): 17.98 (1187 through 66 games)
2015, second (13-20): 16.28 (456 through 28)
2015, third (21-28): 15.07 (422 total through 28 games)

My ability to input things into a calculator is probably slightly faulty, so take these with a grain of salt, but at a cursory glance it does not appear the tossup conversion as a whole went down from last year (if anything, it maybe went up about a third of a tossup per game).
Max
formerly of Ladue, Chicago

User avatar
No Rules Westbrook
Auron
Posts: 1222
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by No Rules Westbrook » Tue Apr 21, 2015 6:26 pm

As to the differences in style between my history and Marshall's history, that's not a debate that's going to get solved here.

As people well know, my writing style is all about coming straight at you with a succession of discrete clues that can be memorized, if you want to memorize them. That certainly doesn't mean you have to memorize them - you can acquire the information in any number of ways, as quizbowl players demonstrate all the time when answering these questions. Marshall is at the other end of the spectrum in terms of style, tending to focus on very conceptual and contextual clues that resist direct identification (i.e. they resist direct one-to-one matching of "clue or phrase x = answer y"). Now, you can be a fan of whatever style you want, or prefer somewhere in the middle (realistically, none of us are completely on one side - we all use clues from both categories, because you have to use the clues that are available, but you can lean one way or another in your construction of questions).

Both styles can produce good questions if they're employed by a skillful writer, and both styles can produce bad questions if they're employed by a poor writer. I think the full Marshallian style may have more pitfalls to watch out for, because there are certain very restrictive confines that the game of quizbowl has to operate under - you can't ask people to buzz in and compose a 4-page essay on the socioeconomic impact of Hideyoshi's sword hunt. But, it can certainly be done well. Even if it is done well, it doesn't mean everyone is going to like it - one style or another may be your cup of tea.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
vcuEvan
Auron
Posts: 1086
Joined: Mon Feb 23, 2004 5:49 pm
Location: Richmond VA

Re: ACF Nationals 2015 Discussion

Post by vcuEvan » Tue Apr 21, 2015 6:50 pm

No Rules Westbrook wrote:Well, here's what I was trying to argue with my post.

If you want to look at a packet where 8 questions went dead in a round in the third bracket and say "that's too many" - then I'm pretty sympathetic to your argument. I was reading in the top bracket, but I do know what it's like in the other brackets, and I'm sensitive to that - especially for a team that's really trying its butt off, but too many questions are going dead to have a competitive match.

But, instead of constantly focusing on that one balls-hard tossup on the Selous Scouts, let's focus on the 7 other questions that went dead in that packet and see what the deal is with those. As Ike said very well, my attitude when writing those small handful of tossups on stuff like Billy Sunday was "I'm fine if this doesn't get answered...it's one of the handful of hardest tus in the event, and if it goes dead, it goes dead." (not anything like "take this, suckers!" or "watch how crazy this is!")

Like, really, go and look at my list of history tossup answer choices - and let's talk about the brunt of those. Not two or three outliers. A lot of my arguing in this thread is because, on the whole, I find the vast majority of those answer choices to be quite tame and responsible (at the very least in the sense of being solidly below what I'd consider "open-level" difficulty)
I agree with you that it's much more important to look at the overall difficulty than the really hard outliers. I even agree that a smattering of outlier difficulty tossups is desirable, as long as the answers are worthy. Unfortunately there's still a problem with difficulty. Here are some of the packets:

Equiano, Piast, Louis the Pious, Spring and Autumn
Pendleton, Goa, Philby's ring, Silla Kingdom
Secretary of Interior, Medizing, Elizabeth of Russia, Kushan
Oberlin, Black and Tans, Licinius, Abydos
Cyrus Vance, Fashoda Incident, Isidore of Seville, Miguel Hidalgo

In all five of these packets, there's 1 answer (Secretary of the Interior) that could comfortably fit into ACF Fall. The rest are a few comfortable Regionals questions, and a ton are things that could only be in Nationals or would be really hard tossup answres at Regionals. You're right that this is the true source of excessive difficulty for most of the teams.

Beyond any philosophical advantages it has, Ted's "core" philosophy does a wonderful job of avoiding this kind of difficulty that leads to 10 tossups being answered between inexperienced teams and isn't revealed by focusing on outlier answers.
Evan Adams
VCU '11, UVA '14, NYU '15

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ACF Nationals 2015 Discussion

Post by grapesmoker » Tue Apr 21, 2015 6:52 pm

I'd say both Fashoda and Hidalgo could plausibly be harder-level Fall answers. Hidalgo in particular seems quite famous to me, though that might be a product of my California schooling. I do agree with the larger point Evan is making, though. A lot of those answer choices are quite rough for the lower brackets.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
No Rules Westbrook
Auron
Posts: 1222
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by No Rules Westbrook » Tue Apr 21, 2015 6:59 pm

Double also, on Max's post, while I'm no math wizard...

I have taken time to run the Nationals stats from 2012 to 2015 just about eight ways from Billy Sunday, and I don't think you can make any kind of convincing empirical argument that this Nationals was harder (not that more tossups went dead, or that less bonus points were scored, or that less tossup points were scored, or that there was a more adverse impact on lower brackets, or anything)

It's just in the zeitgeist now that lots of people think there needs to be a second revolution of new difficulty contraints in quizbowl. And naturally, if that's the way your beard points anyway, this tournament will become the most handy exhibit you can use to promote that agenda. Whatever. I'm not signing the petition, but it's no skin off my back if that's the way it ends up going.

But, if you want to convince me that this set was actually harder, show me some numbers that matter.


coda: Spring and Autumn isn't Fall-level too?'
double coda: almost every single one of those answers is a submitted question. so I'm using submissions. I suppose you're advocating for just replacing submissions with easier editor tossups. you could easily, though, just be the change you want to see - send me really good easy-answer questions and I promise I'll use them.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ACF Nationals 2015 Discussion

Post by grapesmoker » Tue Apr 21, 2015 7:08 pm

The submission point is salient: it is neither possible nor desirable for an editor to try and replace all submissions. If you want to see an easier tournament, write easier questions.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
vcuEvan
Auron
Posts: 1086
Joined: Mon Feb 23, 2004 5:49 pm
Location: Richmond VA

Re: ACF Nationals 2015 Discussion

Post by vcuEvan » Tue Apr 21, 2015 7:11 pm

grapesmoker wrote:The submission point is salient: it is neither possible nor desirable for an editor to try and replace all submissions. If you want to see an easier tournament, write easier questions.
Sure, but a reasonable editor could also take a common link submission on "Pendleton" in American history and decide not to turn it into a tossup on a fourth tier Civil War general that largely went unanswered.
Evan Adams
VCU '11, UVA '14, NYU '15

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ACF Nationals 2015 Discussion

Post by grapesmoker » Tue Apr 21, 2015 7:13 pm

vcuEvan wrote:
grapesmoker wrote:The submission point is salient: it is neither possible nor desirable for an editor to try and replace all submissions. If you want to see an easier tournament, write easier questions.
Sure, but a reasonable editor could also take a common link submission on "Pendleton" in American history and decide not to turn it into a tossup on a fourth tier Civil War general that largely went unanswered.
Well, you'll get no argument from me there.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
No Rules Westbrook
Auron
Posts: 1222
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by No Rules Westbrook » Tue Apr 21, 2015 7:28 pm

Sure, but a reasonable editor could also take a common link submission on "Pendleton" in American history and decide not to turn it into a tossup on a fourth tier Civil War general that largely went unanswered.
I'm certainly not going to die on the hill defending that tossup idea. It's super hard and he's an incredibly minor dude, but for the love of god, can people stop talking about Pendleton and the Selous Scouts. They're two questions.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
magin
Yuna
Posts: 952
Joined: Fri Oct 27, 2006 5:50 pm
Location: College Park, MD

Re: ACF Nationals 2015 Discussion

Post by magin » Tue Apr 21, 2015 7:29 pm

I think the "send in easier questions if you want easier questions" argument is a bit of a copout. Yes, in some sense the submissions guide how the tournament will look, and you want to reward teams for writing good questions by altering them as little as possible. But if you're an editor for a national tournament, you have the final responsibility to control how the set looks, not the teams. Some teams will always send in difficult questions for nationals, either because they want to write on something new and exciting, or they have a difficult pet topic that they want to write about, or because they think that questions for nationals should be hard. It's the editor's responsibility to keep difficulty in line by making sure that overall, most teams can answer most tossups and that tossups have a reasonable amount of middle clues.

To my school of thought, you earn the right to include some harder questions in your tournament by working as hard as possible to ensure that the vast majority of tossups are gettable. Hard tossups are like desserts; it's tempting for editors to include a lot of them. But if you do, you don't end up providing a very nutritious meal.
Jonathan Magin
Montgomery Blair HS '04, University of Maryland '08
Editor: ACF

"noted difficulty controller"

User avatar
vcuEvan
Auron
Posts: 1086
Joined: Mon Feb 23, 2004 5:49 pm
Location: Richmond VA

Re: ACF Nationals 2015 Discussion

Post by vcuEvan » Tue Apr 21, 2015 7:38 pm

No Rules Westbrook wrote:
Sure, but a reasonable editor could also take a common link submission on "Pendleton" in American history and decide not to turn it into a tossup on a fourth tier Civil War general that largely went unanswered.
I'm certainly not going to die on the hill defending that tossup idea. It's super hard and he's an incredibly minor dude, but for the love of god, can people stop talking about Pendleton and the Selous Scouts. They're two questions.
My point with this example is that you can't rest on the "submissions are hard" defense while turning a Fall level answer into maybe the hardest one at the tournament.
Evan Adams
VCU '11, UVA '14, NYU '15

User avatar
Ike
Yuna
Posts: 917
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by Ike » Tue Apr 21, 2015 7:43 pm

Okay, so let's rehash some of these difficulty arguments -

You have one poster coming and here and saying that this tournament was too hard because 6 tossups went dead against UMD / Chicago and this tournament was too hard, but that's kind of anomalous in the sense that Michigan's team plus their opponent answered 18, while in my room, UCSD and I think Minnesota answered 16. If you listen to the podcast, you'll hear that some of the dead tossups are derps - "systems of logic" and accidentally botching "oberlin." I personally was flabbergasted that people don't know what the word "inflection" is but that may be a pocket in my knowledge. The point is using one packet to say the whole tournament is too hard is horrendously wrong extrapolation at its finest.

We then use statistical evidence to show that our tournament is actually easier by ppb, than every ACF Nationals since four nationals ago, but then some of you guys want to use dead tossups instead.

Max then compiles evidence showing that our tournament is actually easier if you do a dead tossup count. And then it still doesn't really settle you guys - you guys claim "oh there aren't enough ACF Fall" level tossups in the history, so this tournament is too hard. To use the vernacular, that's silly. You can write a perfectly legitimate tournament where the easiest answers are from all ACF Regionals. I don't care if my easy lit in the packet is "Baudelaire" or "Victor Hugo" - it's still easy. Billy didn't choose to write on ACF Regionals answerlines or ACF Fall answerlines - he just wrote on whatever was the right answer to convey the clues, and to be honest so did I - I had a bunch of things I wanted to write about on game theory, for example, that people could know about so I just chose it as my answer line.

Since I got home, people who don't post have told me that this was the best tournament they had ever read / played. So maybe I should stop being a bit defensive? I don't know but even when someone like Jacob Reed complains about difficulty and I listen to two podcasts where he's playing games in the second bracket, and in both games he and the other team combines for a total of 19 and 18 tossups - it just doesn't feel right to me, given that we have better bonus conversion and tossup conversion than last year across the top 3 brackets.
Ike
UIUC 13

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ACF Nationals 2015 Discussion

Post by grapesmoker » Tue Apr 21, 2015 7:55 pm

magin wrote:I think the "send in easier questions if you want easier questions" argument is a bit of a copout. Yes, in some sense the submissions guide how the tournament will look, and you want to reward teams for writing good questions by altering them as little as possible. But if you're an editor for a national tournament, you have the final responsibility to control how the set looks, not the teams. Some teams will always send in difficult questions for nationals, either because they want to write on something new and exciting, or they have a difficult pet topic that they want to write about, or because they think that questions for nationals should be hard. It's the editor's responsibility to keep difficulty in line by making sure that overall, most teams can answer most tossups and that tossups have a reasonable amount of middle clues.
This is a lovely sentiment in theory that founders immediately on the rocks of actual practice. ACF Nationals has 25 packets; it's unreasonable to expect an editor to rework every question, especially if it's a sound question with the only fault being that it's hard. If we held people to this standard we'd either have to drastically scale back the tournament or have submission deadlines six months in advance.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
No Rules Westbrook
Auron
Posts: 1222
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by No Rules Westbrook » Tue Apr 21, 2015 8:04 pm

I just did a calculation - in the team packets, 78% of the questions used were submitted, 22% were by me.

The history tossups in those packets written by me are: Costa Rica, New Jersey, Chakri Dynasty, Know-Nothing Party, Goa, Alben Barkley, Metaurus River, Pontiac's Rebellion, Pierre Laval, Pope Urban II, Louisiana Territory

Religion tossups by me: minarets, Tracts for the Times, New Year festivals

Yeah, that's a real murderer's row of tossups there, huh?



I'm undoubtedly being too defensive in this thread too, at least partly because I'm extremely egotistical about the method and research that I use to write questions. However, anyone who's followed my writing career knows that I'm not unresponsive to valid criticisms; I have varied the style, content, and difficulty of my writing a lot over the years to meet the needs of particular events.

That said, I'm not going to sit back and accept as "fact" that this tournament was qualitatively harder in some way than past tournaments, when the evidence shows it just simply was not. I'm not going to kowtow to a false perception. I've been around the block - I know what's hard, what's kind of hard, and what isn't hard. I've written entire tournaments myself that were premised solely on the concept of being hard, and were happily advertised as such - I'm not going to pretend this tournament was in that vein, because it just was not.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
Periplus of the Erythraean Sea
Auron
Posts: 1881
Joined: Mon Feb 28, 2011 11:53 pm
Location: Falls Church, VA

Re: ACF Nationals 2015 Discussion

Post by Periplus of the Erythraean Sea » Tue Apr 21, 2015 8:17 pm

Looking at Max's tossup conversion statistics, I apologize and retract my statement that this tournament had too many dead tossups at the top of the field - answers were hard but clearly the first and second brackets were able to roll with them. As someone who claims to know econometrics, that's pretty embarassingly poor extrapolation on my part, and I apologize for engaging in it. I'll stand by my statement about the choice of hard answers for the lower portions of the field.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor

User avatar
Cheynem
Sin
Posts: 6514
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: ACF Nationals 2015 Discussion

Post by Cheynem » Tue Apr 21, 2015 8:23 pm

I'm sympathetic to the plight of lower brackets; having been on such teams myself, it can be frustrating and annoying to have many dead tossups and unanswered bonus parts. That said, I do wonder to what extent ACF Nationals can fully differentiate between some of the fourth and fifth bracket teams. By this, I do not mean that such teams should not come to Nats and that the majority of games they will play at Nats do not succeed in making such differentiation (I think they do), but that there is the probability that in a non negligible number of games in these lower brackets, some muddy battlefields might emerge. I do agree that probably reducing difficulty slightly across the board would help, but to some extent, yes, fourth/fifth bracket games at very hard sets are going to get messy at times (again, I would certainly be in the fifth bracket now if I chose to play ACF Nationals without teammates to coattail ride, so this is definitely not me saying "learn more things like me!").
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
The King's Flight to the Scots
Auron
Posts: 1442
Joined: Mon Jan 26, 2009 11:11 pm

Re: ACF Nationals 2015 Discussion

Post by The King's Flight to the Scots » Tue Apr 21, 2015 8:23 pm

Mewto55555 wrote:Playoff tossups answered per game, presented without significant elaboration:

(all stats from http://www.hsquizbowl.org/db/tournament ... /playoffs/ and http://www.hsquizbowl.org/db/tournament ... /playoffs/)

2014, top bracket (1-12): 17.15 (1132 through 66 games)
2014, second bracket (13-24): 15.77 (1041 through 66 games)

2015, top (1-12): 17.98 (1187 through 66 games)
2015, second (13-20): 16.28 (456 through 28)
2015, third (21-28): 15.07 (422 total through 28 games)

My ability to input things into a calculator is probably slightly faulty, so take these with a grain of salt, but at a cursory glance it does not appear the tossup conversion as a whole went down from last year (if anything, it maybe went up about a third of a tossup per game).
Max, out of curiosity, wouldn't this analysis underweight the playoff packets that were only heard in the top bracket?

EDIT: No, never mind, I think you accounted for that.
Matt Bollinger
UVA '14, UVA '15
Communications Officer, ACF

User avatar
Mewto55555
Tidus
Posts: 708
Joined: Sat Mar 13, 2010 9:27 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by Mewto55555 » Tue Apr 21, 2015 8:39 pm

The King's Flight to the Scots wrote:
Mewto55555 wrote:Playoff tossups answered per game, presented without significant elaboration:

(all stats from http://www.hsquizbowl.org/db/tournament ... /playoffs/ and http://www.hsquizbowl.org/db/tournament ... /playoffs/)

2014, top bracket (1-12): 17.15 (1132 through 66 games)
2014, second bracket (13-24): 15.77 (1041 through 66 games)

2015, top (1-12): 17.98 (1187 through 66 games)
2015, second (13-20): 16.28 (456 through 28)
2015, third (21-28): 15.07 (422 total through 28 games)

My ability to input things into a calculator is probably slightly faulty, so take these with a grain of salt, but at a cursory glance it does not appear the tossup conversion as a whole went down from last year (if anything, it maybe went up about a third of a tossup per game).
Max, out of curiosity, wouldn't this analysis underweight the playoff packets that were only heard in the top bracket?

EDIT: No, never mind, I think you accounted for that.
I don't really know what this question means, but it did prompt me to look at how the average number of questions varies in each of the top bracket games by packet:

Editors 1: 19.3
Editors 2: 19.7
Editors 3: 18.6
Dartmouth: 18.8
Editors 4: 16.8
Editors 5: 17
Penns & Kenyon: 17
VCU & Berkeley: 18.5
Editors 6: 15.83

I'd be careful about reading too much into this due to small sample size (Editors 4, for example was dragged down massively thanks to a 13 tossup effort from Columbia v. Stanford B, which matched up two of the bottom teams in the top bracket, and Editors 1-3 all saw a #3 seed matched with a #1 seed, so every game had a #1 or #2 seed playing, making less things go dead than when there were #3 v. #3 games), but it does support everyone's complaint that Editors 6 was randomly hard out of nowhere (Editors 6 was so hard it caused the top bracket teams to play like they were approximately the 20th best team in the country on a normal packet).
Max
formerly of Ladue, Chicago

User avatar
magin
Yuna
Posts: 952
Joined: Fri Oct 27, 2006 5:50 pm
Location: College Park, MD

Re: ACF Nationals 2015 Discussion

Post by magin » Tue Apr 21, 2015 8:58 pm

grapesmoker wrote:
magin wrote:I think the "send in easier questions if you want easier questions" argument is a bit of a copout. Yes, in some sense the submissions guide how the tournament will look, and you want to reward teams for writing good questions by altering them as little as possible. But if you're an editor for a national tournament, you have the final responsibility to control how the set looks, not the teams. Some teams will always send in difficult questions for nationals, either because they want to write on something new and exciting, or they have a difficult pet topic that they want to write about, or because they think that questions for nationals should be hard. It's the editor's responsibility to keep difficulty in line by making sure that overall, most teams can answer most tossups and that tossups have a reasonable amount of middle clues.
This is a lovely sentiment in theory that founders immediately on the rocks of actual practice. ACF Nationals has 25 packets; it's unreasonable to expect an editor to rework every question, especially if it's a sound question with the only fault being that it's hard. If we held people to this standard we'd either have to drastically scale back the tournament or have submission deadlines six months in advance.
I don't think this is exactly right, Jerry. Sure, I don't mean that editors have to completely rewrite all of the submissions. What I mean is that editors are receiving a whole bunch of hard submissions, they should rework the answer lines of a number of them to produce a smoother tournament. Speaking from experience, it's not that hard to turn hard tossups into easier ones. Just from past ACF Nationals I've edited, I turned a submission on the poet James Wright into a tossup on authors with the last name "Wright," a tossup on completing The Art of the Fugue into a tossup on The Art of the Fugue, and so on. All you need to do is fine tune difficulty here and there.
Jonathan Magin
Montgomery Blair HS '04, University of Maryland '08
Editor: ACF

"noted difficulty controller"

User avatar
The King's Flight to the Scots
Auron
Posts: 1442
Joined: Mon Jan 26, 2009 11:11 pm

Re: ACF Nationals 2015 Discussion

Post by The King's Flight to the Scots » Tue Apr 21, 2015 9:01 pm

Mewto55555 wrote:
The King's Flight to the Scots wrote:
Mewto55555 wrote:Playoff tossups answered per game, presented without significant elaboration:

(all stats from http://www.hsquizbowl.org/db/tournament ... /playoffs/ and http://www.hsquizbowl.org/db/tournament ... /playoffs/)

2014, top bracket (1-12): 17.15 (1132 through 66 games)
2014, second bracket (13-24): 15.77 (1041 through 66 games)

2015, top (1-12): 17.98 (1187 through 66 games)
2015, second (13-20): 16.28 (456 through 28)
2015, third (21-28): 15.07 (422 total through 28 games)

My ability to input things into a calculator is probably slightly faulty, so take these with a grain of salt, but at a cursory glance it does not appear the tossup conversion as a whole went down from last year (if anything, it maybe went up about a third of a tossup per game).
Max, out of curiosity, wouldn't this analysis underweight the playoff packets that were only heard in the top bracket?

EDIT: No, never mind, I think you accounted for that.
I don't really know what this question means, but it did prompt me to look at how the average number of questions varies in each of the top bracket games by packet:

Editors 1: 19.3
Editors 2: 19.7
Editors 3: 18.6
Dartmouth: 18.8
Editors 4: 16.8
Editors 5: 17
Penns & Kenyon: 17
VCU & Berkeley: 18.5
Editors 6: 15.83

I'd be careful about reading too much into this due to small sample size (Editors 4, for example was dragged down massively thanks to a 13 tossup effort from Columbia v. Stanford B, which matched up two of the bottom teams in the top bracket, and Editors 1-3 all saw a #3 seed matched with a #1 seed, so every game had a #1 or #2 seed playing, making less things go dead than when there were #3 v. #3 games), but it does support everyone's complaint that Editors 6 was randomly hard out of nowhere (Editors 6 was so hard it caused the top bracket teams to play like they were approximately the 20th best team in the country on a normal packet).
My question was dumb.

One thing I'd like to add: Editors' 6 wasn't just one packet, it was an incredibly important packet for the purposes of the tournament. At a quizbowl national tournament, there are usually around 4 teams with a >10% chance of winning. There are going to be three packets where those teams are all playing each other, which all have a huge influence on which teams make the finals. The only one of the top four seeds that lost to any other team was, well, Virginia.

So, when Ryan says that "The only real empirical support [that the tournament was hard] is that the top four teams had a pretty bad packet on the final round EDITORS 6 [and the finals]" - that's actually a really big deal! Editors 6 and the finals aren't just two of 20-some packets, they're two of the four packets that, together, did about 90% of the work determining who won the title.
Matt Bollinger
UVA '14, UVA '15
Communications Officer, ACF

User avatar
No Rules Westbrook
Auron
Posts: 1222
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by No Rules Westbrook » Tue Apr 21, 2015 9:06 pm

Okay, I have no sympathy for a very good quizbowl team whining about Editors Six.

Here's the tossups in Editors' Six: Fifth Column, Candida, Baudolino, malaria, Oberlin College, Black and Tans, Licinius, Abydos, cross section, arrythmia, metric space, coupling reactions, the Seven Sages, Baba Yaga's hut, A System of Logic, Delacroix, concerto grosso, Great Britain, inflection, Green River


There's just nothing difficult about that packet, if you're a good team. You don't have any cause to cry about the overall difficulty of that packet - pretty much every packet at an open tournament is easily that difficulty. Chicago and Maryland had a bad packet on it; get over it. Michigan had a great packet on it. So what?
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
Mewto55555
Tidus
Posts: 708
Joined: Sat Mar 13, 2010 9:27 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by Mewto55555 » Tue Apr 21, 2015 9:14 pm

No Rules Westbrook wrote:Okay, I have no sympathy for a very good quizbowl team whining about Editors Six.

Here's the tossups in Editors' Six: Fifth Column, Candida, Baudolino, malaria, Oberlin College, Black and Tans, Licinius, Abydos, cross section, arrythmia, metric space, coupling reactions, the Seven Sages, Baba Yaga's hut, A System of Logic, Delacroix, concerto grosso, Great Britain, inflection, Green River


There's just nothing difficult about that packet, if you're a good team. You don't have any cause to cry about the overall difficulty of that packet - pretty much every packet at an open tournament is easily that difficulty. Chicago and Maryland had a bad packet on it; get over it. Michigan had a great packet on it. So what?
The "so what" is that it turned out that this packet, regardless of whether or not there's "nothing difficult" about it (a fact I can't speculate about -- I'm really only qualified to discuss the science, which, unlike a lot of the rest of this packet, actually got converted), had a significantly lower conversion rate: in the 6 rooms featuring the 12 best teams in the country, there were 14, 14, 16, 17, 16, 18 questions answered. The round before had 20,16,18,19,19,19 tossups converted. That's a huge difference consistent across the entire bracket. I have no idea whether or not that was the intention of the editors, but it's definitely how the questions played out.
Max
formerly of Ladue, Chicago

User avatar
Nabonidus
Wakka
Posts: 113
Joined: Sat Apr 19, 2014 9:32 pm

Re: ACF Nationals 2015 Discussion

Post by Nabonidus » Tue Apr 21, 2015 9:21 pm

No Rules Westbrook wrote: Here's the tossups in Editors' Six: Fifth Column, Candida, Baudolino, malaria, Oberlin College, Black and Tans, Licinius, Abydos, cross section, arrythmia, metric space, coupling reactions, the Seven Sages, Baba Yaga's hut, A System of Logic, Delacroix, concerto grosso, Great Britain, inflection, Green River
I can't comment on the difficulty but it does seem a little bit odd for there to be three questions on health conditions.

EDIT: Apparently George Bernard Shaw likes to write plays that share their names with pathogens.
Last edited by Nabonidus on Tue Apr 21, 2015 9:52 pm, edited 1 time in total.
Derek So
McGill

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5662
Joined: Mon Oct 23, 2006 2:00 am

Re: ACF Nationals 2015 Discussion

Post by theMoMA » Tue Apr 21, 2015 9:35 pm

The fact that one packet of the editors' packets was particularly unforgiving, as played, is really tough to account for as editors. I'd venture that, if you posted similar lists of answers from one editors' packet to the next, it would be close to impossible to pick out which ones actually turned out to be more difficult. From a practical standpoint, as I'm sure everyone knows, editors' questions get written in chunks, and there's a lot of chance that goes into how exactly they get clumped together into packets. A particularly hard editors' packet is almost certainly just a difficult-to-foresee outlier. So while the effect of Editors' 6 being particularly tough may have been large, it's hard to blame the editors too much for not being able to predict that that would be the case.

(I am much more sympathetic to the argument that finals packets should stop being so hard, because those packets being hard is a very foreseeable consequence of people chucking much of the experimental fare into one packet, not a random constellation of harder stuff in one editors' packet. I really strongly believe that writers should aim for there to be no difference between their regular editors' packets and the finals packets that will decide who the champion is.)
Andrew Hart
Minnesota alum

User avatar
The King's Flight to the Scots
Auron
Posts: 1442
Joined: Mon Jan 26, 2009 11:11 pm

Re: ACF Nationals 2015 Discussion

Post by The King's Flight to the Scots » Tue Apr 21, 2015 9:38 pm

theMoMA wrote:The fact that one packet of the editors' packets was particularly unforgiving, as played, is really tough to account for as editors. I'd venture that, if you posted similar lists of answers from one editors' packet to the next, it would be close to impossible to pick out which ones actually turned out to be more difficult. From a practical standpoint, as I'm sure everyone knows, editors' questions get written in chunks, and there's a lot of chance that goes into how exactly they get clumped together into packets. A particularly hard editors' packet is almost certainly just a difficult-to-foresee outlier. So while the effect of Editors' 6 being particularly tough may have been large, it's hard to blame the editors too much for not being able to predict that that would be the case.
I don't think it's that hard to foresee and account for. If you know that the late editors' packets are going to be the really important ones, look at them beforehand and move stuff around so that they're not oppressively tough. In the tournaments I've worked on, like PACE and NASAT, it's been common practice to take extra care looking at the Superplayoff packets to make sure they're good.

For what it's worth, we set up the VCU Open spreadsheet a couple weeks ago with a system to avoid such clustering. We'll see how it goes in practice.
Last edited by The King's Flight to the Scots on Tue Apr 21, 2015 9:40 pm, edited 1 time in total.
Matt Bollinger
UVA '14, UVA '15
Communications Officer, ACF

User avatar
Sima Guang Hater
Auron
Posts: 1849
Joined: Mon Feb 05, 2007 1:43 pm
Location: Philadelphia, PA

Re: ACF Nationals 2015 Discussion

Post by Sima Guang Hater » Tue Apr 21, 2015 9:39 pm

No Rules Westbrook wrote:Okay, I have no sympathy for a very good quizbowl team whining about Editors Six.

Here's the tossups in Editors' Six: Fifth Column, Candida, Baudolino, malaria, Oberlin College, Black and Tans, Licinius, Abydos, cross section, arrythmia, metric space, coupling reactions, the Seven Sages, Baba Yaga's hut, A System of Logic, Delacroix, concerto grosso, Great Britain, inflection, Green River

There's just nothing difficult about that packet, if you're a good team. You don't have any cause to cry about the overall difficulty of that packet - pretty much every packet at an open tournament is easily that difficulty. Chicago and Maryland had a bad packet on it; get over it. Michigan had a great packet on it. So what?
So I want to take a little bit of a middle ground on this packet. Between UVA and Penn (statistically, the two strongest teams at this tournament), Abydos, Fifth column, malaria, and inflection went dead, malaria only due to dumbassery on both our parts. Additionally, Seven Sages was a total guess on Mattbo's part, as indicated by his reaction after he got it. So by an arguable metric, there were four questions that were above our pay grade. Of these:

1. Abydos is a perfectly good idea for a question - it's an important site in Egyptian archeology, has come up with some frequency, and is the name of the planet visited in the landmark film of our time, Stargate.
2. I'm not qualified to argue about the merits of Fifth Column, so I won't make any pronouncements on it.
3. Inflection went dead, despite Saajid having taken intro linguistics (I'm sure Will got this).
4. Seven Sages seems incredibly minor; our team has two players that are decent at Asian history, and UVA's no slouches either. I was later told by Sarita (who was spectating the game) that they'd covered them in a Chinese history class, so it seems defensible at the very least.

I think Ryan's right in that our low conversion on this packet was more bad luck than anything - it hit the corners of the canon that we randomly don't know very well. I'm guessing Michigan did not have this problem - they lucked out in the other direction. What I think Ryan's wrong about is the fact that this 'bad luck' is actually a decent argument for making the packets a little easier - that way, there's way less of a chance for this kind of game to happen. I remember Ryan's 2008 Chicago Open, where he "knew" Selene and I were going to be in the final - and consequently made the biology tossup so hard that neither Selene or I could get it. Given these kinds of examples, I think it's perfectly reasonable to slightly lowball the difficulty of a given packet to make sure that it's still answerable by top teams.
Eric Mukherjee, MD PhD
Washburn Rural High School, 2005
Brown University, 2009
Medical Scientist Training Program, Perelman School of Medicine at the University of Pennsylvania, 2018
Intern in Internal Medicine, Yale-Waterbury, 2018-9
Dermatology Resident, Vanderbilt University Medical Center, 2019-

Member Emeritus, ACF
Member, PACE
Writer, NAQT, NHBB, IQBT

"The next generation will always surpass the previous one. It's one of the never-ending cycles in life."

User avatar
The King's Flight to the Scots
Auron
Posts: 1442
Joined: Mon Jan 26, 2009 11:11 pm

Re: ACF Nationals 2015 Discussion

Post by The King's Flight to the Scots » Tue Apr 21, 2015 9:42 pm

The Quest for the Historical Mukherjesus wrote:
No Rules Westbrook wrote:Okay, I have no sympathy for a very good quizbowl team whining about Editors Six.

Here's the tossups in Editors' Six: Fifth Column, Candida, Baudolino, malaria, Oberlin College, Black and Tans, Licinius, Abydos, cross section, arrythmia, metric space, coupling reactions, the Seven Sages, Baba Yaga's hut, A System of Logic, Delacroix, concerto grosso, Great Britain, inflection, Green River

There's just nothing difficult about that packet, if you're a good team. You don't have any cause to cry about the overall difficulty of that packet - pretty much every packet at an open tournament is easily that difficulty. Chicago and Maryland had a bad packet on it; get over it. Michigan had a great packet on it. So what?
So I want to take a little bit of a middle ground on this packet. Between UVA and Penn (statistically, the two strongest teams at this tournament), Abydos, Fifth column, malaria, and inflection went dead, malaria only due to dumbassery on both our parts. Additionally, Seven Sages was a total guess on Mattbo's part, as indicated by his reaction after he got it. So by an arguable metric, there were four questions that were above our pay grade. Of these:

1. Abydos is a perfectly good idea for a question - it's an important site in Egyptian archeology, has come up with some frequency, and is the name of the planet visited in the landmark film of our time, Stargate.
2. I'm not qualified to argue about the merits of Fifth Column, so I won't make any pronouncements on it.
3. Inflection went dead, despite Saajid having taken intro linguistics (I'm sure Will got this).
4. Seven Sages seems incredibly minor; our team has two players that are decent at Asian history, and UVA's no slouches either. I was later told by Sarita (who was spectating the game) that they'd covered them in a Chinese history class, so it seems defensible at the very least.

I think Ryan's right in that our low conversion on this packet was more bad luck than anything - it hit the corners of the canon that we randomly don't know very well. I'm guessing Michigan did not have this problem - they lucked out in the other direction. What I think Ryan's wrong about is the fact that this 'bad luck' is actually a decent argument for making the packets a little easier - that way, there's way less of a chance for this kind of game to happen. I remember Ryan's 2008 Chicago Open, where he "knew" Selene and I were going to be in the final - and consequently made the biology tossup so hard that neither Selene or I could get it. Given these kinds of examples, I think it's perfectly reasonable to slightly lowball the difficulty of a given packet to make sure that it's still answerable by top teams.
I guess I'd add that when we looked it up later, "Roman Fever" never actually says Roman Fever is malaria, and Daisy Miller only mentions the word "malaria" I think once. We should have gotten it anyway, but you do need non-trivial outside knowledge to get that one.
Matt Bollinger
UVA '14, UVA '15
Communications Officer, ACF

User avatar
No Rules Westbrook
Auron
Posts: 1222
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by No Rules Westbrook » Tue Apr 21, 2015 9:44 pm

You don't think I wrote that c-myc tossup, do you? Someone else did, and I had no idea how hard it was. I'd certainly retired from science writing by then, after the Great Diol Incident of Nationals.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
Sima Guang Hater
Auron
Posts: 1849
Joined: Mon Feb 05, 2007 1:43 pm
Location: Philadelphia, PA

Re: ACF Nationals 2015 Discussion

Post by Sima Guang Hater » Tue Apr 21, 2015 9:50 pm

No Rules Westbrook wrote:You don't think I wrote that c-myc tossup, do you? Someone else did, and I had no idea how hard it was. I'd certainly retired from science writing by then, after the Great Diol Incident of Nationals.
It was C-Reactive protein at CO. At ACF Nationals it was c-myc, which is actually a pretty easy answerline, I was just not entirely with it.
MattBo wrote:I guess I'd add that when we looked it up later, "Roman Fever" never actually says Roman Fever is malaria, and Daisy Miller only mentions the word "malaria" I think once. We should have gotten it anyway, but you do need non-trivial outside knowledge to get that one.
It said Ronald Ross too
Last edited by Sima Guang Hater on Tue Apr 21, 2015 10:23 pm, edited 1 time in total.
Eric Mukherjee, MD PhD
Washburn Rural High School, 2005
Brown University, 2009
Medical Scientist Training Program, Perelman School of Medicine at the University of Pennsylvania, 2018
Intern in Internal Medicine, Yale-Waterbury, 2018-9
Dermatology Resident, Vanderbilt University Medical Center, 2019-

Member Emeritus, ACF
Member, PACE
Writer, NAQT, NHBB, IQBT

"The next generation will always surpass the previous one. It's one of the never-ending cycles in life."

User avatar
No Rules Westbrook
Auron
Posts: 1222
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by No Rules Westbrook » Tue Apr 21, 2015 10:01 pm

Yeah, let's just be clear - I'm not gonna give anyone flack for arguing that finals should not throw difficulty into overdrive mode like that. I've always been a fan of ultra-hard Finals, but it's always a gamble. I certainly think there's a very cogent argument against it.

Any editors six complaints, though, will be immediately filed in the special pleading mailbox.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
The King's Flight to the Scots
Auron
Posts: 1442
Joined: Mon Jan 26, 2009 11:11 pm

Re: ACF Nationals 2015 Discussion

Post by The King's Flight to the Scots » Tue Apr 21, 2015 10:11 pm

No Rules Westbrook wrote:Yeah, let's just be clear - I'm not gonna give anyone flack for arguing that finals should not throw difficulty into overdrive mode like that. I've always been a fan of ultra-hard Finals, but it's always a gamble. I certainly think there's a very cogent argument against it.

Any editors six complaints, though, will be immediately filed in the special pleading mailbox.
Special pleading? You're the one who asked for THE NUMBERS, over and over again, for like six straight posts. Then you got them for Editors Six. Now you're saying they don't matter because you don't like what they say.
Matt Bollinger
UVA '14, UVA '15
Communications Officer, ACF

User avatar
Ike
Yuna
Posts: 917
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by Ike » Tue Apr 21, 2015 10:14 pm

For editors six, I definitely did not try to make my questions harder.
I remember Ryan's 2008 Chicago Open, where he "knew" Selene and I were going to be in the final - and consequently made the biology tossup so hard that neither Selene or I could get it.
One thing I'd like to add: Editors' 6 wasn't just one packet, it was an incredibly important packet for the purposes of the tournament. At a quizbowl national tournament, there are usually around 4 teams with a >10% chance of winning. There are going to be three packets where those teams are all playing each other, which all have a huge influence on which teams make the finals. The only one of the top four seeds that lost to any other team was, well, Virginia.
Matt and Eric - I agree with what you're saying - I told Billy to make sure those science tossups were absolutely pristine for the editor's packets and the three team packets that we had. In that sense I agree.

At the same time, I actually took a very "randomized approach" to how I wrote the editor's packets: every time I wrote a tossup for a category, I used random.org to fill that tossup randomly into one of the remaining spots in the editor's packets. I would have done the same thing for the finals, but Ryan informed we were going to do a harder finals, so I didn't want to have my categories be of a different flavor than Ryan's, Rob's and Billy's. I think this is important for two reasons: one, you don't do stupid shit to rig a match like "man i want to see Team Michigan win, so I'm going to put this linguistics or whatever tossup in a packet where they are going to be playing UVA / UMD," and two, you don't let your personal feelings on what's important decide individual games: e.g. "In order to win a finals, you must absolutely know about game theory." That's why epokhe ended up in packet 1 - because it was actually truly random, even though, as a harder tossup, it would have been "nicer" if it were in editor's 6 or something. The Veil of Ignorance at work here!

Matt, there's no guarantee that Penn, UVA, UMD and Chicago would have been one seeds - although it is likely of course. You could have faced UMD or Chicago if they accidentally threw a game in the prelims or whatever (look what happened with UMD Columbia.)

To be honest, I love the idea of using team packets for playoff games and making it so that we don't know which packets are used where except for the finals - I don't like the idea of editor's automatically assuming that their packets are superior and that they will be used in the playoffs since superior teams will be in the top bracket. In fact, if I remember correctly there was this fiasco about editors last year wanting to run the playin-final game with a packet that was already read because they thought it had "cool questions" or whatever. That's seriously fucked up - edit your questions so that team packets are cool and don't try to "rig matches" with your own cool questions.

Ike
Ike
UIUC 13

Jason Cheng
Rikku
Posts: 361
Joined: Wed Jan 18, 2012 3:23 am

Re: ACF Nationals 2015 Discussion

Post by Jason Cheng » Tue Apr 21, 2015 10:49 pm

The Quest for the Historical Mukherjesus wrote: 4. Seven Sages seems incredibly minor; our team has two players that are decent at Asian history, and UVA's no slouches either. I was later told by Sarita (who was spectating the game) that they'd covered them in a Chinese history class, so it seems defensible at the very least.
At the risk of wading into this discussion without knowing how to swim, I want to say that the Seven Sages of the Bamboo Grove seem analogous to some Roman mystery. They're interesting and I think they're well-studied in academia from a Jin-ish era cultural history perspective (Daoism, art, philosophy, counter-Confucianism), and they're a pretty common theme in Chinese art. That said, I'm actually extremely bad at real Chinese history and was only aware of them because I like walking around in museums whenever I'm in Asia.
Jason Cheng
Arcadia High School 2013
UCSD 2017
PACE
http://www.socalquizbowl.org

Lightinfa
Wakka
Posts: 109
Joined: Tue Apr 21, 2009 9:21 pm
Location: Richmond, VA

Re: ACF Nationals 2015 Discussion

Post by Lightinfa » Tue Apr 21, 2015 11:04 pm

The problem here is not just one of the questions being hard- it's for lack of a better term that they were a bad kind of hard. I'm a history player who enjoys hard questions as much as the next guy, and in a secondary role I'd like to think I scale up well enough at this difficulty level- usually being able to scrape out 10-15 ppg on good teams. That's not what happened here- from the second half of a day one onward, I think I managed all of two tossups. This wasn't because the tournament was "too hard," rather lots of things I have deep knowledge of simply did not show up. For better or for worse, I'm entirely dependent on "real" knowledge from reading books and classes, and that knowledge was for the most part useless as we were faced with tossup after tossup focusing on dynastic politics, rulers, and battles. Obviously those things are important, but written in such a way as to acknowledge their role in the larger historical context and narratives to allude to Matt's quote of Chris is the other thread. It seems to me that the more difficulty increases, the more people rely on their reservoirs of real knowledge, and this tournament did not scale up in a way that rewarded how people gain real, deep history knowledge.

This problem was compounded by really weird choices in the history sub-distribution. For instance, we for some reason got two tossups on eighteenth century Russian rulers (Catherine and Elizabeth) and not a single tossup on World War I, World War II, or the Cold War! Vastly important areas and time-periods just did not show up in the tossups, like modern Asia. We got hard tossups on peripheral time periods and locales instead of hard tossups on areas that get a lot of study. I know we've focused an inordinate amount of attention on the Pendleton tossup, but that's a good example of how this tournament increased difficulty the wrong way- instead of writing a hard Civil War tossup on one of the very many important Civil War figures quiz bowl never asks about (say for instance Nathaniel Banks, on whom plenty of people are going to have real knowledge), we got a hard tossup on an unimportant figure where if he's converted at all it's unlikely it gets beyond "have you heard of him" bowl.
JR Roach
Maggie Walker, 2011

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ACF Nationals 2015 Discussion

Post by grapesmoker » Tue Apr 21, 2015 11:46 pm

Let me try to refocus my feelings about this set into a set of easily-digestible points:

1. I think overall this was a very high quality set.
2. It does not seem that the data we currently have available supports the assertion that this set was unusually difficult in aggregate compared to the last 3 years of ACF Nationals. After the bloodbath of 2011, the next 4 Nationals look, without doing any actual analysis, to be pretty similar in terms of overall difficulty.
3. There were things in this set that seemed excessively difficult without any particularly good reason. This came through in some questionable answer choices on tossups and extremely difficult hard bonus parts.
4. The set could have been made a bit more accommodating to the lower-bracket teams by easing up on the bonus difficulty and cutting one or two of those really hard tossups without sacrificing any ability to distinguish between top-bracket teams.

That last point is what I really want to focus on. Mike asked a relevant question about how to distinguish top-bracket teams. I maintain that the best way to do so is with the hard bonus parts. In many instances of this tournament, I felt that the bonus was basically a maximum 20 because there was no way anyone was coming up with the hard part. That's not good even if the aggregate bonus conversion is more or less fine; a good team should have a legitimate shot at 30 points, and too often I think that was not the case. Actual numbers won't be available until next week, but I do believe that the top 3 or 4 teams should be able to top 20 PPB, and that the margin between them should primarily be established by their ability to convert hard parts.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
Sima Guang Hater
Auron
Posts: 1849
Joined: Mon Feb 05, 2007 1:43 pm
Location: Philadelphia, PA

Re: ACF Nationals 2015 Discussion

Post by Sima Guang Hater » Wed Apr 22, 2015 12:48 am

Can the packets be posted soon?
Eric Mukherjee, MD PhD
Washburn Rural High School, 2005
Brown University, 2009
Medical Scientist Training Program, Perelman School of Medicine at the University of Pennsylvania, 2018
Intern in Internal Medicine, Yale-Waterbury, 2018-9
Dermatology Resident, Vanderbilt University Medical Center, 2019-

Member Emeritus, ACF
Member, PACE
Writer, NAQT, NHBB, IQBT

"The next generation will always surpass the previous one. It's one of the never-ending cycles in life."

User avatar
Ike
Yuna
Posts: 917
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: ACF Nationals 2015 Discussion

Post by Ike » Wed Apr 22, 2015 1:13 am

I disagree with Jerry's theoretical foundations. In no way, do I think that "3 or 4 teams breaking 20ppb" is ideal - in fact, I think fitting your bonus to some type of ideal distribution, to say "this is what an ideal tournament is like" is a mistake on many levels, for one you basically have to guarantee that your medium parts are super easy for top teams: like that Samuel Slater medium part has to be made way easier because Chicago didn't get it! Jonathan Magin, noted difficulty controller, thought it was a fine medium part - but under this rather draconian model of difficulty, he's very wrong. In fact, Jerry's model can be achieved by doing this: make tossups worth 20 points, and have two part bonuses: the first part should be converted by 10 out of the 12 top teams and the second part should be converted by the top 4 teams some percentage of the time. If you think about it - this model of quizbowl fits Jerry's distribution for the top team bracket - you get 10 bonus points since top teams should be perfunctorily converting easy parts! I hope most people see that this format causes more weight to be placed on tossups, rather than actually placing weight on bonuses - which is ironic, since Jerry claims to be advocating for "hard parts to distinguish teams."

I'd also like to point out that at no ACF Nationals, has this "goal" ever come closed to being achieved. In fact, In the last ten years, this year came very close to having 4 teams break 20 ppg - the only one that is arguably closer is 2014. Why all of a sudden this tournament caused people to foist this standard is a little bit beyond me - especially since we were arguably closer than most other tournaments.

Let me instead offer my own picture of bonus difficulty: I write on a topic and you get a value in between 0 and 30 points for your knowledge of the topic! As an example, consider my Jarman / Olivier / merchant-ivory bonus - intended as h/m/e - This was 0'd by Stanford A and presumably would have been 30'd by UVA or Michigan. I have no problem handing out a 0 because if you don't know who Olivier is - you don't know film. That's how you figure out who is a national champion - you give 30 points to UVA for knowing a fuck ton of film and 0 points to teams that don't know it. It's certainly not "Oh Stanford is a top bracket team - you must give them 19 points on average, per a bonus!"

I know that example and approach seems callous, but honestly it's actually awesome for two reasons: One, you actually distinguish between teams with all three parts: For some reason, people nowadays assume that because two teams are top teams, they must have "deep or similar knowledge" of every subject, which is so dumb, one top team can have great knowledge of film and another can have near zero. - I could have made that Olivier part so that "90% of teams got that bonus part by having it be Shakespeare" but honestly, what is the point of that? We're not distinguishing any knowledge of the subject at hand!

Secondly, and I think equally important, is that it is rewarding when any team gets 10, 20 or 30 bonus points on a topic. I know it may not feel like it given today's aesthetics of quizbowl, but teams in the bottom bracket should be ~proud of themselves~ for getting 10 points on any bonus at this tournament! Knowing who William Hogarth or Sir Laurence Olivier is means that you are a step above most other intelligent people! But, if you write it so that "90-100% of teams" will convert the easy part or something, you're just going to have to make all your bonus parts like Easter Bunny stupid.

In fact, let me offer my theory about how we reach out to middle-bottom bracket teams: You write bonuses that they can conceivably 30 and will feel good about 30ing: every team has someone with a deep pocket knowledge in some category: a 10 ppg player in the bottom bracket might be super knowledgable in jazz, math, or some other random category. You focus on writing that good tossup that rewards knowledgable people for their random pockets of academic knowledge by tapping into areas that quizbowl doesn't usually tap into - you certainly don't do it by making your easy parts dumbly easier so that "all teams can convert it" - in fact that makes the battlefield at the bottom muddier.
Ike
UIUC 13

User avatar
vinteuil
Auron
Posts: 1320
Joined: Sun Oct 23, 2011 12:31 pm

Re: ACF Nationals 2015 Discussion

Post by vinteuil » Wed Apr 22, 2015 1:29 am

Ike, I think you know that Jerry is arguing for averages to try to conform to some standard, not every single bonus. Similarly, I don't think he's saying that easy parts need to be perfunctory.
Jacob Reed
Chicago ~'25
Yale '17, '19
East Chapel Hill '13
"...distant bayings from...the musicological mafia"―Denis Stevens

User avatar
vinteuil
Auron
Posts: 1320
Joined: Sun Oct 23, 2011 12:31 pm

Re: ACF Nationals 2015 Discussion

Post by vinteuil » Wed Apr 22, 2015 1:33 am

Also, I think that dead tossups as a difficulty metric might be a tiny bit misleading; this tournament also had at least what felt like a lot of questions likely to be converted only near the end, e.g. Memling's Last Judgment (when the Memling tossup last year was probably hard enough!), which presumably had an easy enough giveaway involving souls being weighed or something.
Jacob Reed
Chicago ~'25
Yale '17, '19
East Chapel Hill '13
"...distant bayings from...the musicological mafia"―Denis Stevens

User avatar
Auroni
Auron
Posts: 2988
Joined: Thu Nov 15, 2007 6:23 pm
Location: Brooklyn

Re: ACF Nationals 2015 Discussion

Post by Auroni » Wed Apr 22, 2015 1:39 am

Ike wrote: Secondly, and I think equally important, is that it is rewarding when any team gets 10, 20 or 30 bonus points on a topic. I know it may not feel like it given today's aesthetics of quizbowl, but teams in the bottom bracket should be ~proud of themselves~ for getting 10 points on any bonus at this tournament! Knowing who William Hogarth or Sir Laurence Olivier is means that you are a step above most other intelligent people! But, if you write it so that "90-100% of teams" will convert the easy part or something, you're just going to have to make all your bonus parts like Easter Bunny stupid.
Okay, I'm going to jump in here because you just made an argument that bothers me. It seems like people always make a false dichotomy between the current versions of too-hard easy parts and "find your ass" bonus parts like the Easter Bunny. In fact, there's plenty of room to have easy parts that present a substantially lower barrier for entry to a bonus than George Telemann, Tristan Tzara, or Italian kings named Umberto, without doing something like dipping into asking for the countries that those people were from given their capitals, and it makes a tangible positive impact on a huge swath of the field for little to no editorial time investment. Why wouldn't you do it?
Last edited by Auroni on Wed Apr 22, 2015 1:43 am, edited 2 times in total.
Auroni Gupta
UIUC

Locked