Feedback about Feedback

Elaborate on the merits of specific tournaments or have general theoretical discussion here.
Post Reply
User avatar
CPiGuy
Tidus
Posts: 603
Joined: Wed Nov 16, 2016 8:19 pm
Location: Ann Arbor, MI

Feedback about Feedback

Post by CPiGuy » Sun Dec 09, 2018 8:47 pm

Hi, quizbowl!

There's been a lot of discussion about the current state of tournament feedback in the Discord, and I want to offer my thoughts on how we can make tournament feedback better. This applies to both high school and college tournaments, but unfortunately forum posts cannot live in two subforums at once.

The primary problem with quizbowl tournament feedback, as I see it, is the "silence-as-compliment paradigm" -- that is, the vast majority of tournament feedback concerns problems that people had with the tournament, and I have seen people being advised to consider a lack of feedback as a sign that their questions were good. I am not at all saying that people shouldn't give negative feedback -- they absolutely should! -- but people who found a set or question to be particularly good should not hold back from saying so on the forums! It's just as important for writers to know which of their questions were particularly good (as opposed to merely unremarkable), as it is to know which of them were bad. It is also demoralizing for writers to experience only negative feedback for questions that they put a lot of time and effort into crafting.

Another problem is that there isn't enough feedback. Offering feedback on sets is something that more people should do! To use a personal anecdote, I wrote about 35/35 for 2017 NASAT. It was the first tournament I ever wrote for that wasn't a vanity math tournament. I was really interested to hear what people thought of my questions, so I made a post in the specific discussion thread detailing the questions I was responsible for and asking for feedback. I got zero feedback on my questions, and there were maybe two posts in the entire thread that weren't just people asking to see a specific question. This might be an extreme case, but it was pretty disappointing.

On the note of "people asking to see a specific question", I understand that in many cases an obstacle to providing detailed feedback posts is that, unlike me, most quizbowlers do not write down almost every answerline in a set, and therefore have a hard time remembering details. I often have a hard time remembering things even with that, and often make notes in my notebook of particularly good or bad questions to remind myself to comment on them later. To that end, I think that tournament producers who are interested in receiving feedback should try to make copies of their set available to those who have played the tournament, and should be proactive about this -- perhaps posting a link to the set, or at the very least a list of answerlines, in the private discussion forum. (I understand that there are some question security issues, but every advanced-stats tournament I've played has done this as a consequence of using advanced stats, and it has not compromised the security of those tournaments and had a noticeable impact in people's ability to discuss the set.)

The third problem is that of giving feedback in the wrong places. Expressing your opinions of a set on the public Discord, whether positive or negative (but especially if negative) is not productive -- you literally cannot defend those opinions without citing unclear set content, so you can't really have a meaningful discussion of how to improve the set. The Discord is a great place to have a lot of productive conversations. Discussion of unclear set content is not one of them. In a particularly egregious example of this that occurred today, someone accused an unclear high school set of plagiarism without any significant evidence (I was made privy to the questions involved, which while they clued many of the same things did so in a pretty different order and with different phrasing in every line but the giveaway). This is a really serious accusation to levy in a public forum where the accusee cannot defend themselves, especially when it's not at all clear that the accusation is correct! (To that person's credit, when this was pointed out, he retracted those accusations.) Don't do this, people! If you have opinions of a set, you need to either bring them up with the editors directly, or use a space -- like the private discussion forums -- where you can actually meaningfully discuss the set without spoiling question content.

TL;DR: Private discussion forums for sets exist. Use them!
Conor Thompson
Bangor HS (Maine) '16
Michigan '20

User avatar
Skepticism and Animal Feed
Auron
Posts: 3170
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: Feedback about Feedback

Post by Skepticism and Animal Feed » Wed Dec 12, 2018 10:58 am

You're going to get more bang for your buck if you give high level, systemic feedback to a question writer or editor.

If you tell a question writer "your middle clues were too hard, here is some advice on how to write good middle clues", that is something they can apply to every question they ever write in the future.

If you tell a question write "well, your question on Basil the Bulgar-Slayer was not pyramidal because you mentioned the Battle of Kleidion before the Battle of the Gates of Trajan, and the Battle of Kleidion is way more famous. We learned about it in my 8th grade Bulgarian history class even!", that feedback may be objectively correct and warranted, but the writer is only going to be able to apply that feedback to Basil the Bulgar-Slayer tossups, and how many of those are they going to write again in the future? Maybe three if they write an absolute ton of history tossups?

If you think about systemic feedback you can give, and not just specifics about each tossup you didn't like, you might prevent 50 future bad tossups instead of just 3 future bad tossups!
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source

alexdz
Rikku
Posts: 381
Joined: Wed Sep 03, 2008 7:29 pm
Location: Philadelphia, PA

Re: Feedback about Feedback

Post by alexdz » Wed Dec 12, 2018 11:37 am

What Bruce said is absolutely correct. I will add/clarify that you should absolutely address any *factual* errors at the individual question level so that the editors have a chance to correct them for any future mirrors (and for the sake of improving their knowledge). But Bruce is right that beyond the level of such factual mistakes, the high-level, big-picture feedback is more important and more applicable.
Alex Dzurick
====
Owner/Editor, SAGES Quizbowl Questions
Coach, Harcum College (PA)
====
Former midwesterner (South Callaway - Mizzou - UIUC) coping with life on the east coast.

User avatar
Deviant Insider
Auron
Posts: 4531
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland

Re: Feedback about Feedback

Post by Deviant Insider » Tue Dec 25, 2018 11:09 am

Like Conor, I wish I received more feedback on my questions. It might be good to talk about things writers can do to get more feedback.

First of all, there are some good reasons why just opening up a private forum does not usually get that many responses:
1) Quizbowl people generally are busy, and they are especially busy around a quizbowl weekend. After a tournament, a priority for a lot of people is getting back to real life.
2) Here is how an average player experiences an average tossup: They hear about four lines of text of clues they do not recognize and that are read quickly, and then somebody else buzzes in with the correct answer. It is difficult to give feedback on that text. Also, I think people are hesitant to say that some bonus was on something they didn't know much about, so the bonus appropriately asked a middle and hard part that they got wrong.
3) A lot of people are not on the forums.
4) A lot of people don't have strong opinions about questions. They just think of questions as things that appear at quizbowl tournaments.
5) It is often difficult to know where to start. People want to give general criticism, but they often have strong reactions to individual questions that don't fit into some pattern.

When people play or moderate a set, their natural reaction is to focus on the material they are strongest in because that is where they are able to judge leadins, hard parts of bonuses, whether the clues were interesting, whether the answers were appropriate, and so forth. However, when there are a small number of commenters, and the commenters are focused on niche subjects, the writers and editors often don't get the feedback they were looking for. A disproportionate part of the discussion may be on something that was a small part of the distribution that the writers/editors weren't trying to do anything original with. (To be clear, this is an explanation for why feedback can be frustrating. It is not at all a call for people to stop commenting on their favorite subjects, which people are very welcome to do.)

I think that there are some things that writers/editors can do to get more feedback than just opening up a private forum. For the sake of openness, I'll admit that I don't do these things.
1) If there are writers and editors you respect who aren't playing the tournament, try to bring them on board while the set is being written.
2) If there are writers and editors you respect who aren't working with you for whatever reason, contact them directly and seek feedback, preferably using some of the other points below. If they play the set, then of course you have to do this after the tournament.
3) Pick out a handful of questions you would like feedback on, and post them in the forum and include them in any message soliciting feedback.
4) If there was something in particular you were trying to do differently in this set, be very explicit about that and ask for feedback about it. Even if you weren't trying to do something differently but it is some issue you think about when writing/editing, then be explicit when asking for feedback.
5) Announce when you post a set and be explicit that you want feedback from people who study it.

I think we as a community should be clear about the fact that it is perfectly fine to give feedback on a set whenever you get around to reading it, even if that is six months or two years after the set was first played. Also, initial feedback does not have to wait--you can give feedback when you hear a set and then give more/different feedback when you study it.

To be clear, nothing in this post negates anything Conor (and Bruce and Alex) said. I agree with them.
David Reinstein
Head Writer and Editor for Scobol Solo and Masonics (Illinois), TD for New Trier Scobol Solo and New Trier Varsity, Writer for NAQT (2011-2017), IHSSBCA Board Member, IHSSBCA Chair (2004-2014), PACE Member, PACE President (2016-2018), New Trier Coach (1994-2011)

User avatar
UlyssesInvictus
Yuna
Posts: 784
Joined: Thu Feb 10, 2011 7:38 pm

Re: Feedback about Feedback

Post by UlyssesInvictus » Tue Dec 25, 2018 2:16 pm

Random idea that popped into my head, but have any writers tried just explictly emailing a Google form to the coaches/teams after a tournament and asking for feedback? (Or if a mirror site, asking the hosting site to do so on their behalf?)

Just something with standard questions about difficulty, packetization, specific errata, etc, and maybe a provision about whether the form answerer would allow their response to be x-posted on the forums to spur further discussion.

If anyone has tried this, I'd be curious about results, and if they haven't, I'd be curious as to any potential downsides.
Raynor Kuang
quizdb.org
Harvard 2017, TJHSST 2013
I wrote GRAPHIC and FILM

jonah
Auron
Posts: 2293
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: Feedback about Feedback

Post by jonah » Tue Dec 25, 2018 3:32 pm

UlyssesInvictus wrote:
Tue Dec 25, 2018 2:16 pm
If anyone has tried this, I'd be curious about results, and if they haven't, I'd be curious as to any potential downsides.
Question security, for one thing. If the form really does come from someone who should be allowed to see details about question content and is similarly limited in who can see the results, it's fine, but that's hard to establish conclusively, and it is easy for people to fall into bad habits about question security, or (especially for less-connected people) to just not think about it at all.
Jonah Greenthal
National Academic Quiz Tournaments

User avatar
Progcon
Tidus
Posts: 550
Joined: Fri Dec 20, 2013 8:24 pm

Re: Feedback about Feedback

Post by Progcon » Tue Dec 25, 2018 3:35 pm

UlyssesInvictus wrote:
Tue Dec 25, 2018 2:16 pm
Random idea that popped into my head, but have any writers tried just explictly emailing a Google form to the coaches/teams after a tournament and asking for feedback? (Or if a mirror site, asking the hosting site to do so on their behalf?)

Just something with standard questions about difficulty, packetization, specific errata, etc, and maybe a provision about whether the form answerer would allow their response to be x-posted on the forums to spur further discussion.

If anyone has tried this, I'd be curious about results, and if they haven't, I'd be curious as to any potential downsides.
I did this for the Spartan Housewrite Discord mirror. The tournament was only a couple of days ago, so I haven't got a ton of responses, but I am very grateful for what I have gotten so far. I had the feedback Google form be optional and if a team decided to fill it out, they would get a 10 dollar rebate.
Harris Bunker
Grosse Pointe North High School '15
Michigan State University '19 B.S. in Economics & B.A. in Mathematics
UC San Diego Economics 2019 -

"Whereof one cannot speak, thereof one must be silent."- Ludwig Wittgenstein

matthewspatrick
Lulu
Posts: 30
Joined: Tue Jul 15, 2014 11:30 am
Location: Wilmington, DE

Re: Feedback about Feedback

Post by matthewspatrick » Tue Jan 01, 2019 9:42 am

jonah wrote:
Tue Dec 25, 2018 3:32 pm
UlyssesInvictus wrote:
Tue Dec 25, 2018 2:16 pm
If anyone has tried this, I'd be curious about results, and if they haven't, I'd be curious as to any potential downsides.
Question security, for one thing. If the form really does come from someone who should be allowed to see details about question content and is similarly limited in who can see the results, it's fine, but that's hard to establish conclusively, and it is easy for people to fall into bad habits about question security, or (especially for less-connected people) to just not think about it at all.
One simple way to manage concerns about security: paper. Indeed, this could be extended to asking for mod/scorekeeper feedback from each round. (Given that the experience level of officials at regular invitationals is likely to be lower and more variable at invitationals, feedback for them may be even more valuable than it is at the nationals level, for which officials have been much more closely vetted.)
Patrick Matthews
University of Pennsylvania 1989-94
NAQT Member Emeritus and co-founder

Post Reply