On in-game feedback in quizbowl

[Gautam]

Hi quizbowl,

This past weekend, I staffed CO. I haven't really staffed too many "high profile" collegiate events like this in the past... this was also the first event with a truly 'national' participant body that I've staffed since ICT '13, and the first tournament that I staffed since MUT '11 for which I had also edited a lot of questions.

While reading, or observing as the bye-teams were reading packets, I heard a lot of the statements that we have come to associate with these "high-profile" events. These are outwardly expressed sentiments, usually of frustration, that abound once the doors are closed and the tossup-bonus cycles have begun.

I think we've had a few threads about what makes for good feedback on the forums, and in all the threads, we've come to an agreement that first impressions are worthless. I'd like to request to everyone in the community that you apply the same recommendations to the in-game feedback that they give.

Being loud is anti-productive

The current system we have where some people are really loud in passing judgements is anti-productive. It bothers the other team if they have a shot at the remainder of a tossup/bonus, or just slows down the pace of a game. But, more importantly, the loudness that is prominent in an in-game situation rarely materialize as a concrete point of discussion on the forums. Similarly, in all the years that I've edited questions, I've never had a player e-mail me (or even just come up to me and strike a conversation) with thoughts on a particular question, either.

This is basically an indication to me that most of the in-game feedback that is spewed about is something you the player don't actually care about, and that I should ignore it. And that's what I have been doing, recently. So, if you want to help me other editors become better at incorporating this "real-time" feedback, then I strongly suggest you start getting better about the feedback you give. If you continue to make snap-judgements, I am going to have to ignore them unless you e-mail me things if it's about individual questions, or write a post about your on the forums if it was a trend in many questions.

Things like "that was really hard" or "that was an interesting idea" are okay, IMO. But again, please elaborate on the forums on in-person about what you thought was too hard. Jerry had a good example for this for re: some physics bonuses - he told me that some bonuses was too hard, and then mentioned specific examples in the CO discussion thread, and why he thought they were hard. This is the kind of feedback that is helpful.

Being loud can be insulting
Very often, the in-game feedback uses third-person pronouns to refer to the writers and editors of the questions who are present in the room. "why did THEY do that?!" or "that was stupid of THEM" are altogether too common. This use of third-person pronouns irks me... it suggests that the editors are some mythical set of people too distant or busy with other things to hear the feedback that players have to give. I am speculating here, but I suspect this may have been the perception, for instance, of NAQT of years past... when SCT/ICT editors didn't make it onto the forums as much, and were probably not super in-touch with the circuit.

It is simply not true today. All your editors have the quizbowl savvy to lurk around and/or comment on the forums, make it into the IRC, play-test tournaments they don't plan on playing, stay in-tune with the circuit by attending tournaments, etc. Your editors are not some distant third-party dictating tossups to their secretaries. So stop referring to your editors as such.

To give you an example of how this can be done well - Stephen Liu walked up to me after the end of a game in which there was (moot) protest on a physics question, and asked me what I, as a science editor, thought of the protest. I had to respectfully decline to comment, since I knew very little about the topic at hand. But I made sure to follow up with the person who did know, and passed along the feedback on how the question played out. If Stephen had only grumbled and groaned after the tossup and walked away, I basically wouldn't have bothered following up.

There are also cases where the feedback refuses to acknowledge that there was some editor putting in a lot of effort to craft good questions and making conscious decisions about his product. There was an egregious example I saw when there was a perfectly fine ecology bonus that was read. At the end of the bonus, a couple of people erupted into "Oh my god, Nick Jensen questions!"

I found this insulting on 3 counts.

It was insulting to the original writer, who was present in the room, who came up with the idea for the bonus.
It was insulting to the editor, who was present in the room, who solidified what was an interesting submission into a well thought-out bonus. Nick had nothing to do with the editing of the bonus beside a comment he left about the fact that he liked the bonus.
It was insulting to Nick, for perpetuating some kind of wrong stereotype that he writes questions on things only he knows.

I repeat again...

IT WAS INSULTING x 3,

Don't do it. It is maddening as all hell to see that the thoughtful work Gaurav put into crafting that bonus basically resulted in factually incorrect perception like "Haha, we got boned by Nick Jensen's whimsy" rather than "we 0d this bonus because we didn't know ecology."

I'd like to summarize by saying that what you say in an in-game environment provides feedback to the writers and editors of the questions... and if you are not thoughtful, most of what you're saying is going to be relegated as verbal diarrhea rather than some kind of meaningful criticism upon which the writers and editors can act. We have already had a lot of discussion on what makes for good criticism and how it can be helpful. I'd ask you to start incorporating that in your in-game feedback. I can't say I'm completely guilt-free when it comes to practicing what I am preaching here, but I think I've definitely toned it down compared to my collegiate days, and have been following up on my in-game complaints with emails and/or posts in discussion threads. I ask everyone to do the same.

Thanks,
Gautam

[Ike]

So I kind of get this post, but on the other hand too, I get frustration: quizbowl is an intense game played by very intense people and naturally people are going to be angry when they don't get a question. I really wouldn't take it personally if someone insults a question I wrote in the heat of the moment. To be fair, no one did come up to me and say "your question fucking sucked" or anything of that sort, but even if someone did come up to me and say that I'd try to figure out where they are coming from and figure out if they are still mad after 3-4 hours or something. After all, the players of this game invest waking up at 6-7am on probably 5-6 hours of sleep, if even that, to come out and play a game that really has no compensation other than satisfaction - so I'm willing to cut players who give incendiary feedback a lot of slack, especially if the players are remarking in the heat of the moment: that's why we wait until later to get meaningful commentary.

Ike

[Skepticism and Animal Feed]

Let me take this in a behavioral direction by suggesting that a lot of our instant feedback to quizbowl questions might be driven by cognitive biases, and you will be able to provide better feedback once you've calmed down, read the full text of the questions, and seen it in the context of the full tournament.

I think with certain writers/editors, they have a reputation for producing a particular kind of question. And we tend to overestimate the extent to which this is true. For example, I was known to be fond of Hungary. There were in fact side tournaments (most notably Wild Kingdom) where I purposefully played up to this reputation by putting in an egregious amount of Hungary-related clues and answerlines. But for any "serious" tournament, like ACF Nationals, I kept my Ugromania in line. Yet people still complained that there was too much Hungarian content - despite the fact that I looked at the benchmarks and found that my editors questions contained less Hungarian content than the submissions did, and and less or the same than the work of Jerry Vinokurov, an editor with no such reputation. But people were psychologically primed to expect Hungarian content when playing one of my tournaments, and so they remembered the Hungarian tossups at my tournaments but not at Jerry's.

I had a similar experience at the 2010 ICT - I knew that Jeff liked birds, and I started making comments each time a bird-related clue came up. Finally one of my teammates noted to me that the bird content was actually quite low, and when he challenged me to list the actual number of bird clues that had come up I realized, to my surprise, that it was not as many as I seemed to think.

[Skepticism and Animal Feed]

On a different note:

Often, when players express emotions during a tournament, e.g. by throwing things or cursing loudly, they are not necessarily expressing rage at the editor or at the other team, but simply frustration with themselves. As long as quizbowl is a competitive activity, games will be emotional and players will set high expectations for themselves that they often won't meet. If they're just venting about the fact that they didn't get a tossup, as opposed to venting about the tossup or the author, I think it's probably healthy to just let them be.