Where do you think questions come from?

[grapesmoker]

Where do people think questions for events come from? There appears to be some divergence of opinion about this that I've noticed on the circuit within the last two years.

Also, some advice:

Protip #1: don't put your guerilla trash tournaments on the same day as packet-submission tournaments that were announced months in advance.

Protip #2: if you don't plan on coming to the tournament, don't tell the TD you're going to show up and then not show up. Clownboat.

Protip #3: when telling people why you aren't coming to their event, be sure to not let it get out that you'd much rather play pub trivia.

[Rothlover]

I am pretty sure that Jesus told his apostle Steinhice that the Tribute Questions could be found in the mouth of a fish, whereupon they have been feeding lazy programs for nigh upon a decade.

Edit: I should probably note that this was more thatthinginmymouth-in-cheek than a jab at Charlie personally, who I have never had anything but solid dealings with, so I wanted to make clear that I wasn't say, trying to offend him.

I think, in my case, there is just a lot of frustration in seeing two events that people put a lot of time into (and yes, I accept full blame for a lot of the faults with some of my question selection and all of my grammar), that each drew four teams. In each case, efforts were made by numerous people to try and simply get teams there, in the case of the former (our Wirt), I tried to make fees reasonable and offered extreme discounts (and in some cases the chance to play for free,) in the latter, three people, two of whom weren't the editor, were putting in hours doing whatever it took to get teams to an event. It is simply damn frustrating.

[Chris Frankel]

Well, while we're talking about annoying trends in QB as of late, does anyone find it disappointing that it looks like only ~3 independently edited academic packet sets will have been produced this past spring? I miss the days where you could make a schedule of as many local tournaments and still be able to trade for another handful of newly written blind packets for practice purposes.

Also, not to give any of the mirror hosts a hard time, but does anyone also find it annoying that every recent tournament (i.e. BUTT/Wirt) has some insanely convoluted mirror schedule such that you have mirrors spread out across 5 week spans (or in the case of Yale, wasn't it like 4-5 months) so that nobody can engage in any sort of productive public commentary about any tournament until next year when all three dozen spread out mirrors have finished?

I mean, I definitely appreciate maximizing the audience range of the questions, and I understand some scheduling obstacles exist, but it's pretty damn frustrating since the questions at both tournaments have important issues to be discussed for the sake of educating future writers and editors and how to and how not to go about putting together a set, and by the time people are actually authorized to make some sort of commentary it will be when enough time has passed to wipe out or distort their memory of the questions or interest in discussing them. I know that Jerry, Matt, et al are the types of people who solicit and appreciate comments (as am I, since I wrote packets for both events), so I imagine this is annoying from an writer's standpoint too. In any case, in the future, can we try encouraging people to produce more independent question sets or work for more coordination in mirroring so this doesn't become a fact of life?

[Skepticism and Animal Feed]

Because editors write or otherwise obtain (through mirrors or whatever) enough questions to run the tournament anyway, teams will be able to play at a tournament whether or not they actually write questions. So, discounting things like self-improvement or saving a late fee, there's no incentive to write questions. You get the benefit (tournament play) without having to pay any of the costs (time spent researching and writing questions).

[nafai]

You get the benefit (tournament play) without having to pay any of the costs (time spent researching and writing questions).

Bruce: free riding on the backs of hard-working quizbowlers since 2004.

Maybe it's time to prohibit entry to teams consisting of non-novice players who don't submit packets.

--David

[PaladinQB]

Could it be that folks are discouraged from writing questions / running packet-submission tournaments when one of two things happen?

1. Their questions are not used and no feedback on making them better is offered, or

2. Their tournament is ridiculed in this and other forums by vocal circuit personas with very specific ideas of what constitutes good quiz bowl?

bms

[bsmith]

Also, some advice:

Protip #1: don't put your guerilla trash tournaments on the same day as packet-submission tournaments that were announced months in advance.

“Monthsâ€

[grapesmoker]

[quote="bsmith"]“Monthsâ€

[grapesmoker]

Could it be that folks are discouraged from writing questions / running packet-submission tournaments when one of two things happen?

1. Their questions are not used and no feedback on making them better is offered, or

I wonder, what would you have us do? I appreciate the teams that wrote questions for the various packet-submission tournaments that I have edited in the past, but if I let unedited questions get through in their original form, I'd be laughed off the face of the earth. As for feedback, I personally ask (practically beg) for feedback after every tournament I run, and I am always happy to offer feedback to anyone who has questions about why their packet was changed in a particular way. In fact, all experienced editors do this. This should be patently obvious from the discussion following ACF Fall. I personally try my hardest to keep the answer choices that teams select for their packets; if I can improve a question while keeping the answer, I always do that instead of replacing it.

2. Their tournament is ridiculed in this and other forums by vocal circuit personas with very specific ideas of what constitutes good quiz bowl?

If you read the ridicule, you will notice that the people who come under attack are not inexperienced teams writing bad packets, but experienced editors leaving bad packets unaltered. I obviously don't expect a newly-formed team to produce questions of extremely high quality. On the other hand, I expect the editors, who almost always have years of experience under their belts, to improve those questions.

[grapesmoker]

Bruce: free riding on the backs of hard-working quizbowlers since 2004.

Maybe it's time to prohibit entry to teams consisting of non-novice players who don't submit packets.

--David

I think what Bruce means is not that he himself does such things but that there is no obvious incentive to submit packets.

Unfortunately, prohibition would result in fields of zero teams for many tournaments. Personally, after editing a number of packet-submission events, I am beginning to feel that the effort isn't worth it. The whole idea behind the packet-submission event is that the work is distributed across multiple people (as well as to give teams a chance to get better by writing). Just to give you an idea of how easy it is for a team to come up with a packet, consider that a full packet typically consists of 24/24. Dividing that up among 4 people translates to 6/6 per person. Therefore, if you write 1/1 per day (something that should take half-an-hour at most), your team will finish a packet in under a week. Obviously there are exceptions since sometimes teams don't have a full contingent or whatever, it's obvious that for most cases, putting together a full packet is quite easy between three or four people.

[First Chairman]

What we have here is a failure of learning.

1) Does anyone read the stylistic notes on writing questions anymore? I can almost bet you that none of your "problem writers" reads those guidelines.

2) Knowing what makes a clue difficult or easy is always a hard call unless you have experience. Many of the new writers out there don't have a lot of experience, so as expected their questions may need work or outright replacement. There again... experience.

3) Garbage-in, garbage-out. I empathize with the editors out among us (I'm editing a set and taking a break from it) that we get a lot of crap. That happens. I deal with it, and I plan for it. Unfortunately, our sources are all students who do not see benefit from writing questions, even if you give them financial breaks. Honestly, do you believe question writing quality would increase if we said at the next tournament: "submit a quality packet by this deadline, and you will be PAID at least $... higher if the packet is of acceptable or exceptional quality"?

4) Subjective evaluations versus objective evaluations. I think all of us as editors can come up with certain objective evaluations. "This topic belongs in pop culture, not academic" is one example. "You focused way too much on science biography" is another.

I will agree with Sax, if you are to critique, do your best to critique in private. Embarassing the heck out of people in a public forum does no one any good. Of course, if these are experienced editors, I'm sure all of us can take the lumps, though I always know that one may have reasons to leave a bad question in (sure editors are ridiculed but maybe the editor wanted the author to be shown why that question doesn't work).

I really am not sure what to do. It's not like writing a question packet is going to help someone get to graduate quicker. It's not a thesis. It's not a journalistic article for pay. Should we start treating it as such? I don't know, but financial penalties can only go so far.

On the scheduling side of things, that points to a problem in networking. You can claim the date, but keep promoting that date so that people don't take over your particular weekend. Of course, maybe having a calendar would help matters assuming people look at it (like the Yahoo qb group calendar).

[Skepticism and Animal Feed]

The question should be asked: why do people who write questions as of right now write them, despite the fact that it is a seemingly irrational thing to do? (See my post above)

I'm going to go out on a limb and hypothezise that many of them are not motivated by late/early fees at all. Maybe I'm de-sensitized to costs by being the ex-Treasurer of a very well-endowed team (even though we lost Chad Gerson to an admiralty court clerkship), but after spending about $90 for the entry fee and over $100 for travel, does an additional $40 or $15 really make that much of a difference?

Instead, I'm going to propose that a lot of the players who do write questions are motivated by functional incentives. That is, they are motivated by a strong belief in the packet-submission tournament system and a strong desire to see that system continue to exist. From a superficial observation, it seems to me that most of the top players share such a passion, and it wouldn't surprise me if those same top players are also the writers of the majority of packets out there.

I also think that self-improvement can be a strong incentive, mainly because I know from personal experience that the prospect of self-improvement as a player is the major reason why I bother to write questions despite the fact that, under the current system, I don't have to.

If you proliferate those kinds of incentives (don't ask me how), I think you'll get much better results than just toying with late/early fees.

[aescandell]

Therefore, if you write 1/1 per day (something that should take half-an-hour at most), your team will finish a packet in under a week. Obviously there are exceptions since sometimes teams don't have a full contingent or whatever, it's obvious that for most cases, putting together a full packet is quite easy between three or four people.

So I think you're confounding this issue a little bit by breaking it down amongst four people and six days. What you're saying is that it takes twelve man hours to write a packet. Tournament packet penalties rarely amount to over 60 dollars. Maybe people don't like working for five bucks an hour? Raising the packet fee is a good way to get no one to show up. I think reducing the number of required submissions, particularly for tournaments which have a number of mirrored sites and can more easily combine packets, would be a worthy experiment, but I don't really know anything about that.

Also, it might take half an hour to write 1/1 if you write a whole lot and have a bunch of research material on hand, but it probably takes most people longer to write a decent pair of questions. After all, quality standards have risen to the point where every question is expected to come from multiple, varied sources and internet resources are considered invalid. The mere style of writing pyramidal tossups is awkward and leads to difficulties, and many lines equals many clues, each of which must conform to expectations of accuracy, usefulness, and originality. I write only very rarely and sparsely, as it tends to take me an hour or more to come up with a tossup that hints at decency. I dunno what other people's experiences are.

It's worth pointing out that back when packet submission tournaments abounded questions were a lot worse by current standards. Is it possible that the standards of question quality have risen to the point that only the exceptional (in talent or dedication) are capable of producing a significant number of usable questions?

[fizzball]

I'm not saying Cancel Bowl was malicious. I am saying it was careless of them not to make sure nothing else was happening that same date, and also careless not to move their event when it became obvious there was a conflict.

If it was careless for Mark to not pay attention to the goings-on of this forum, do you also acknowledge your own carelessness in not posting your tournament announcement to the Y! board? It is a two-way street, and if this mess has proven anything, it's that both boards remain valuable to the community.

---Greg

[grapesmoker]

So I think you're confounding this issue a little bit by breaking it down amongst four people and six days. What you're saying is that it takes twelve man hours to write a packet. Tournament packet penalties rarely amount to over 60 dollars. Maybe people don't like working for five bucks an hour? Raising the packet fee is a good way to get no one to show up. I think reducing the number of required submissions, particularly for tournaments which have a number of mirrored sites and can more easily combine packets, would be a worthy experiment, but I don't really know anything about that.

I would even say that tournament packet penalties are almost never over $30. If you meet the early deadline, you will probably save $50. Sure, no one likes working for $5/hour, but that's not really what this is about. I certainly don't advocate raising the packet fee; in fact, one of my goals is to make tournament more affordable, not less so. But even with 3 mirror sites, we ran into the lack of questions problem with Aztlan Cup. Only 4 packets of 11 came from people actually playing at any of the sites at all. this is a problem considering that I don't think it's too unreasonable to expect about 4 packets from each site. The pitifully small demand for Brown's event, coupled with extraneous circumstances that two of the early-committed teams had with their packets, made it hard to fill that goal. I don't know if other sites experienced similar problems.

Also, it might take half an hour to write 1/1 if you write a whole lot and have a bunch of research material on hand, but it probably takes most people longer to write a decent pair of questions. After all, quality standards have risen to the point where every question is expected to come from multiple, varied sources and internet resources are considered invalid. The mere style of writing pyramidal tossups is awkward and leads to difficulties, and many lines equals many clues, each of which must conform to expectations of accuracy, usefulness, and originality. I write only very rarely and sparsely, as it tends to take me an hour or more to come up with a tossup that hints at decency. I dunno what other people's experiences are.

I don't think the expectations of most editors are as high as you think they are. I know that I corrected some questions this time around which would have been fine the way they were submitted. Usually I do this if it's a question on my specialty and I see something I think needs tweaking. Other than that, though, as long as a question is accurate and attempts at pyramidality and isn't right out of Wikipedia, it makes most people happy.

I wish I could provide some sort of generalized template for writing tossups. It's hard because questions come from all sorts of different categories, but I'll give it a try anyway:

Consider that there are what I'd call 1st, 2nd, and 3rd order clues, organized in terms of their difficulty. For example, what a critic wrote about a work of art is probably likely to be a 1st-order clue (unless the critic is very, very famous, in which case it could be a 2nd-order). So let's say you wanted to write a tossup about a literary work. Remember, a tossup only needs to be six lines or so. You might start out with references made to the work by other writers or critics. Or perhaps this work influenced someone to write something else, or to base a character on a character in this work. Those could all be 1st-order clues. 2nd-order clues could refer to incidents in the work that are not too well-known, or minor characters. Finally, you could wrap up your question with 3rd-order clues, of the sort where you give the main character and the author of the work.

Obviously, this isn't meant to be a hard-and-fast rule set. It's just a suggestion for ordering your clues that I think can be generalized to other topics. As for Internet sources, keep in mind that many books that are in the public domain are now available for free online from places like Project Gutenberg. Of course, what Random Crazy Website says about a book is not a good thing to put into your question, but most question writers are college students. It should be a simple matter of going to your library and checking out a survey history book, or a science book, or whatever. I do this all the time, especially for things like biology and chemistry which I don't know from a hole in the ground and yet have to write.

It's worth pointing out that back when packet submission tournaments abounded questions were a lot worse by current standards. Is it possible that the standards of question quality have risen to the point that only the exceptional (in talent or dedication) are capable of producing a significant number of usable questions?

I don't think that's true. There were plenty of good tournaments around 2003 and 2004. Take a look at the Cardinal Classic sets from back then, or the WIT sets (those are the ones I have experience with). I think those are good examples of questions produced in adherence with the philosophy of correctness and pyramidality.

The standards have not changed; more questions are being written that don't conform to those standards, and I think that's problematic. Since I want to be constructive, I'm more than happy to share my question writing experience with anyone who is interested. I hope the above comments are useful to you and to other new question writers.

[nafai]

Bruce mentioned in some thread or another that, in high school, teams who were not properly dressed started off at -15 points. Perhaps an in-game penalty for not submitting a packet would be more effective than a financial one. To many, avoiding a loss is more of an incentive than costing their team some money.

[grapesmoker]

I'm not saying Cancel Bowl was malicious. I am saying it was careless of them not to make sure nothing else was happening that same date, and also careless not to move their event when it became obvious there was a conflict.

If it was careless for Mark to not pay attention to the goings-on of this forum, do you also acknowledge your own carelessness in not posting your tournament announcement to the Y! board? It is a two-way street, and if this mess has proven anything, it's that both boards remain valuable to the community.

---Greg

You're absolutely right, it was careless of me not to post on the Yahoo! board. I pretty much assumed that the readership of the two boards was the same and that I could reach anyone who read that board with a post here. But even so, after this happened, I informed Mark, and he chose not to move his event. I understand that there had already been some planning that went into this, but that still doesn't change the fact that pretty much the entire target audience for our event knew about when it was going to happen.

[vandyhawk]

In response to Chris's post about mirrors, we decided in late Feb. that we wanted to host a tourney in March since there was essentially nothing in the southeast between NAQT and Moon Pie. We also knew, though, that given the short notice and other commitments (like our HS tournament this past weekend), there was no way we'd be able to put together a packet submission tournament ourselves - thus the mirror route. We'll probably try to do a packet submission event next year, though given the number of teams registered for our non-submission event this weekend, we'd have to make sure more people would actually come.

As for getting people to write questions, that's been the single most frustrating thing for me running the team the last few years. I don't think we've ever submitted a packet for a "B" team b/c I obviously couldn't take care of it, and the composition of said team has rarely been solidified more than a week prior to a tournament. Even for some people who have played on our A team, it can be quite difficult to get stuff from them on time. I see writing as a basic requirement for participation, whereas a lot of people just see it as a burden and will try to avoid doing it. Not sure how to change that attitude - it's hard to enforce restrictions on people for not writing, since although it could possibly make them more dedicated, it can also alienate them and hurt the team in the long run.

Wasn't there a calendar on the collegequizbowl.org site? The forum seems to be dead, but the domain is still active - anyone know if the calendar is dead too, and if so, whether it could be resurrected?

[Matt Weiner]

After all, quality standards have risen to the point where every question is expected to come from multiple, varied sources and internet resources are considered invalid.

I don't know where that impression comes from. Wikipedia is an unnacceptable source, but the web equivalents of respectable print sources and even some web-only resources like the Genealogical Guide to Greek Mythology site are superb. Especially for the normal invitational level, I imagine it's possible to write half or more of an excellent packet out of Britannica Online.

There's no question that you are right when you say quality standards have risen from where they were ten years ago and this requires more work to go into questions, but on the other hand it seems like everyone who is earnestly interested in learning to meet those standards has been able to do so by their second year of playing, because they accept and incorporate criticism instead of throwing some hissy fit about how us horrible online people are hurting their self-esteem by pointing out how much they suck. It's no secret that the players/teams that are still submitting terrible questions in their fourth, fifth, and sixth years of existence are the ones who took poorly thought out ideological stands against good questions in the first place--i.e., the ones who refused to write good questions. No one smart enough to want to play this game in the first place is incapable of learning to write.

[aescandell]

I should have been clear that I essentially agree with both Jerry and Matt, that question writing is a good thing and more people writing more questions is a good thing. I also agree that questions now are as good or better than they've ever been before, and think still further improvement is possible. My point is simply that question writing is time consuming and difficult, and it's hard to get anyone to do anything time consuming and difficult, regardless of how obviously it might be in their best interest, and if you're not particularly interested in improving as a player, question writing isn't obviously in your best interest. Unfortunately, other than altruistic circuit preserving sorts of goals, it's hard to see how you pitch question writing. All of which I suppose is too obvious for me to have delurked for, but whatever. I was bored.

It's cerrtainly not my intent to excuse people for not writing, nor to whine about it.

[No Rules Westbrook]

The first thing I would say to novice players trying to write good questions as far as structure is to download some more-recent acf nats sets and copy the hell out of them. Answer selection and having the knowledge to figure out which clues go where will still be problems, of course. To reinforce Weiner, there is nothing wrong with internet sources if people know how to do good searches (i.e. ones that avoid wiki and its myriad of myrmidon websites), cross-reference with other sites to verify material and use some common sense, and instead of just copying word for word to actually try to piece together clues.

All this being said, it doesn't matter. I'm convinced that writing questions is an arduous enough task that people who don't feel some sense of loyalty to qb or some pride in furthering the qb circuit (and making themselves better) will never be bothered with writing. The people who would respond to this post by implying that writing questions is equivalent to some 5-buck-an-hour job illustrate the truth of this well enough.

[cvdwightw]

About the mirrors on some extended series of weekends: I expect almost every tournament that has multiple mirror sites in mid-March to have this problem because of conflicting spring break schedules. I would have preferred to push Aztlan Cup back a week or so, but our spring break starts at the end of the last final this week. This meant March 18 was out because with everyone studying for finals we wouldn't have enough staff to run a decent tournament and March 25 is over spring break. For many other schools their break started a week or two earlier than ours does. This conflict makes it darn near impossible to run the same tournament on the same day in March.

As for the other months, I'm not sure why this is. When planning a mirror/packet swap, everyone involved should be able to settle on one date or two consecutive weekends.

Because editors write or otherwise obtain (through mirrors or whatever) enough questions to run the tournament anyway, teams will be able to play at a tournament whether or not they actually write questions.

See, here's the thing. That's wrong. We were worried we wouldn't have enough questions to actually run Aztlan Cup until Jerry sent us his and Matt Weiner's packets at roughly 1 in the morning our time (4 in the morning his time). You can accuse the tournament editors of malicious intent in that only 3 of the 11 packets are editors' packets. Perhaps someone should have stayed up even later (Ray was up until 4:30 in the morning editing the last few packets) and made it a full 1/3 of the packets by editors?

Anyway, the point is that between Jerry and the UCLA editing team we probably spent at least a full packet's worth of questions filling out packets and replacing repeats. And sooner or later, all those freelance packets are going to stop coming in. So then what happens? Are we going to kick teams out of the field last-minute because we only have enough packets to run a 7-team tournament? Are we going to come up with new and convoluted schedules involving partial round robins? I certainly hope not, but I'm not going to pull an all-nighter or go without lunch the day of the tournament just to finish another packet or 2 of questions.

[First Chairman]

After all, quality standards have risen to the point where every question is expected to come from multiple, varied sources and internet resources are considered invalid.

I don't know where that impression comes from. Wikipedia is an unnacceptable source, but the web equivalents of respectable print sources and even some web-only resources like the Genealogical Guide to Greek Mythology site are superb. Especially for the normal invitational level, I imagine it's possible to write half or more of an excellent packet out of Britannica Online.

I also join Matt: I don't know where the quality standards are. I know Matt has a good idea about the quality of questions he's received compared to ones that he's written. I think that every question should come from a verifiable scholarly source. This isn't research in the scholarly "I have to write a thesis" sense.

To that end, would it be a good idea for us (PACE and/or ACF or whoever) to create a webpage with a non-definitive (since that's impossible for the internet) list of web-based resources to write questions? I also think it would be good practice to ask each question author to list their source for their questions.

[bsmith]

To that end, would it be a good idea for us (PACE and/or ACF or whoever) to create a webpage with a non-definitive (since that's impossible for the internet) list of web-based resources to write questions?

A (good?) starting point would be Dwight Kidder's old Reference Desk: http://www.fraughtmachine.com/refdesk/
Peter McCorquodale also keeps a list (with a Canadian slant) as part of his packet guidelines for VETO: http://www.caql.org/events/veto05q.html#Topics

[AE Ismail]

I think saying that Wikipedia is completely unacceptable probably is too harsh. Relying on the Wikipedia as a starting point for ideas for questions is probably useful for beginners. However, you certainly wouldn't want to rely on it as an exclusive source of information. Unless it's something like the spelling of a certain title, who wrote a particular poem, or the year in which an event occurred (basically anything more than refreshing my memory), anything I find in Wikipedia is double-checked against the rest of the Internets.

[/i]

[Matt Weiner]

Unless it's something like the spelling of a certain title, who wrote a particular poem, or the year in which an event occurred (basically anything more than refreshing my memory), anything I find in Wikipedia is double-checked against the rest of the Internets.

It also overemphasizes what is important so it can screw up answer selection and tossup structure even if you do double-check particular facts. And I don't know why you would trust it on facts like common spellings, authors, or years any more than for other purposes; those facts are put there by people from the Internet, usually by plagiarizing a random geocities page, with no citations just like everything else on Wikipedia is.