Faster (Human) Parsing of Answer Lines

Dormant threads from the high school sections are preserved here.
Locked
User avatar
avp
Lulu
Posts: 19
Joined: Sun Nov 09, 2014 11:34 am
Location: Earth

Faster (Human) Parsing of Answer Lines

Post by avp »

While moderating tournaments, you may have come across an answer line that looked something like this example from Chicago Open 2018 found via a quick search on QuizDB. Note that the example was chosen simply for its demonstrative relevance to this post, and certainly not as a means to single out the writers or editors of this particular question.
ANSWER: Scotland [or Alba; accept Scotch Snap; prompt on United Kingdom or U.K. or Britain, but do not accept or prompt on "England"]
This answer line is quite complex to be parsed by the moderator at game speed, due to its length as well as the number of separate clauses, each with their own distinct introductory words. There are many answer lines seen in tournaments that use timed rounds which are significantly more complex than the example above; occasionally, answers are longer than the question itself.

Some tournaments, notably NAQT's newer nationals sets, have chosen to mitigate problems that arise for the moderator by listing these complex answer lines at the front of the packet. This solution, while certainly helpful, has varying degrees of success in practice. The moderator must hold multiple complex answer structures in memory for up to nine or ten minutes. Any answer given by a player must then be reconciled against these structures, often necessitating rereading of the answer line anyway, adding cognitive load for the moderator and slowing the rate of play.

I'd like to discuss ways to increase the readability of the answer lines to make ruling on correctness of complex answer lines simpler, faster, and less error-prone. I have listed two proposals below, in increasing order of invasiveness.

A few notes first:
  • This is not a suggestion to change the substantive content of answer lines, merely the way they are presented to the reader.
  • I am not implying that these ideas are perfect, just providing them as starting points for a discussion in an attempt to make moderators' jobs easier.
1. (Easy) Replace "do not accept or prompt on" with "reject"

We return to our example from above, artificially reformatted for demonstrative purposes:
ANSWER: Scotland [or Alba; accept Scotch Snap; prompt on United Kingdom or U.K. or Britain, but do not accept or
prompt on "England"]
Note the line break before the words 'prompt on "England"'. When ruling on an answer given at game speeds, the answer line as written could be prone to an insidious error.

Suppose a player provides "England" as a response. The moderator, skimming the answer line quickly, may see that phrase and immediately prompt, failing to notice the "do not accept or" preceding it in the answer line. I have made this mistake previously, and surely other moderators have as well.

Additionally, "do not accept or prompt on" is an unnecessarily lengthy phrase.

We can replace the full phrase with "reject". This approach is minimally invasive and provides the benefits of completely avoiding the line break issues in favor of clean yet clear wording which does not risk losing the negation. It also reduces the length of the phrase substantially, making moderator calls faster.

Applying the proposal results in the following slightly more readable answer line:
ANSWER: Scotland [or Alba; accept Scotch Snap; prompt on United Kingdom or U.K. or Britain, reject "England"]
2. (Harder) Improve iteration through answer clauses

Once more, the example answer line:
ANSWER: Scotland [or Alba; accept Scotch Snap; prompt on United Kingdom or U.K. or Britain, but do not accept or prompt on "England"]
A moderator, in the worst case, must read and understand every clause of the answer line before ruling on it. Long lines and non-standard variable length clauses make this either slow or error-prone, particularly in timed rounds. Consider one possible variant on the answer line in which we split it into three separate categories:
ANSWER: Scotland OR Alba OR Scotch Snap
PROMPT: "United Kingdom" OR "U.K." OR "Britain"
REJECT: "England"
This allows the moderator to quickly find the actual answer provided by a player and rule on its correctness. Of course, this format allows for significant flexibility. Each answer possibility could be placed on its own line, though that would seriously increase the amount of paper used for printed questions.

This approach is certainly not without its downsides. Some answer lines would likely take more space than they take right now. On packets intended to only be read electronically, this probably doesn't matter much, but could result in more pages per packet for tournaments which use paper (notably HSNCT and the like). However, this approach provides the greatest benefit when answer lines are already very long, and can be applied selectively at editing time to only certain tossups.

Other Suggestions?

The two items I've listed above are just some basic ideas that I've come up with. If you have other suggestions or ideas to improve answer line readability, please help.
Aakash
SOUNDTRACK, 2018
User avatar
vinteuil
Auron
Posts: 1454
Joined: Sun Oct 23, 2011 12:31 pm

Re: Faster (Human) Parsing of Answer Lines

Post by vinteuil »

I'm intrigued the idea of putting ACCEPT/PROMPT/REJECT on separate lines, but I think it could actually decrease moderator speed. That's because the current best practice is to put "most-anticipated" answers first; sorting by ACCEPT/PROMPT/REJECT means that some very likely answers may end up on the third line that the reader scans, as opposed to being the second entry in the answerline. (Of course, I could be convinced that this philosophy is not actually being implemented in practice.)

While the advent of underlining promptable answerlines and using "quotes" only for rejectable answers ameliorates the potential confusion of "do not accept or / prompt on "wrong answer"," "reject" or a similar term very well might be a good idea in any case.
Jacob R., ex-Chicago
User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2891
Joined: Sun Nov 15, 2009 12:57 am

Re: Faster (Human) Parsing of Answer Lines

Post by Cody »

#1 is mooted by proper underlining: acceptable answers are bolded and underlined, promptable answers are underlined, and unacceptable answers do not have any formatting. This makes it more difficult for moderators to mess up when quickly scanning the answerline, and is more effective than retooling the leading words. (Though shortening them wouldn't go awry.)
Cody Voight, VCU ’14.
Joshua Rutsky
Tidus
Posts: 663
Joined: Mon Oct 24, 2005 9:59 am
Location: Hoover, AL

Re: Faster (Human) Parsing of Answer Lines

Post by Joshua Rutsky »

Admittedly, I’m not in the reading loop at the same level I was once, but I was unaware of the boldface/underline/no format conventions. It might be a good thing for those of us who are once a year hosts or infrequent readers to add a note at the front of the set to this effect.
Joshua Rutsky
VP for Curriculum and Camp Operations, Qwiz
ASCA Board Member
Hoover High School Coach (Retired)
User avatar
Skepticism and Animal Feed
Auron
Posts: 3238
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: Faster (Human) Parsing of Answer Lines

Post by Skepticism and Animal Feed »

Anecdotally, seeing "England" in scare quotes made it really easy for me to understand that it was somehow different from the list of promptable answers in front of it.
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source
User avatar
ValenciaQBowl
Auron
Posts: 2558
Joined: Thu Feb 05, 2004 2:25 pm
Location: Orlando, Florida

Re: Faster (Human) Parsing of Answer Lines

Post by ValenciaQBowl »

Huh--I actually like that a lot. It's a clean look and easier to read.

How would you modify it to include the kind of specific prompting questions that have become more common of late, such as "What class of diseases were treated?" or "Prompt on 'xxxxx' by asking "In what publication?" I guess they could also just come after "Prompt" but then that line could get pretty long.
Chris Borglum
Valencia College Grand Poobah
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: Faster (Human) Parsing of Answer Lines

Post by Mike Bentley »

I think it would be an interesting exercise for someone to do a top-down review of the quizbowl packet / scoresheet from a design perspective. There have been a lot of small "quality of life" improvements to the reading experience in recent years but I bet there are lots more out there if you really put your mind to it. Are there fundamental assumptions we make about how a quizbowl packet should look that should be changed?

I think this would be the most neat if it was in the context of a printed paper packet. In other words, just making changes to wording, layout, fonts, instructions rather than relying on software / hardware.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
Locked