Answerline Selection Analysis

[naan/steak-holding toll]

A lot of folks have been talking recently about how they'd like to see more productive discussions on the forums. Since PIANO is clear now, I wanted to see if I could get something going by posting an analysis I did for a PIANO tossup and see if we could build on it. In particular, I'm curious to see any further analyses on the possibility of technical execution flaws in common-link questions, different answerline choices, etc.

The original post:

Let's take Charlie up on his contention and see if we can learn anything from analyzing the use of the answerline of Polish

Your post had enough other decrees about what is right and wrong about question construction in questions that were absolutely in the range of acceptable that I feel OK saying what I said. Rather than declaring that the Ferrara question "didn't need to exist," or "There's zero reason for this not to be a tossup on Stanislaw Lem*," and perpetuating the norm of people sounding constantly angry over perceived imperfection, is it that hard to instead say "I don't understand what asking about Poland brings to the table, I think a tossup on Lem might have played better?

I'll preface by saying that I think there are three main motivating advantages for using common links:

1) Thematic: A common link can highlight a particular common theme across several works/topics, or within a single topic, that a more standard answerline might not be able to accomplish
2) Accessibility: A common link can allow you to ask material that would be too difficult or otherwise unfeasible to toss up on its own
3) Artistic / Creative: A common link can be on a novel answer which is cool in and of itself, and which is fun to play

I would like to expand on this theory at some point, ideally in a longer and more public post, because there are a lot of drawbacks to common links as well, including leaving a lot more room for technical execution flaws than "standard" tossups. But for now, back to Lem.

Now, this Polish tossup really isn't a common link, since it's only a "link" to one author, but the choice of answerline is clearly motivated using the same line of reasoning - the selection of the answerline Polish was done because Stanislaw Lem is a fairly challenging author who may not get a high conversion rate if asked on his own. Let's try to think of players who benefit from the selection of Polish as an answer versus those who are at a disadvantage with this answer selection (since I suspect this is a non-zero number of people):

Players who are advantaged
- People who recognize clues about Lem and do not remember his name, but do remember that he's a Polish author / wrote in Polish
- People who, at the giveaway, do not recognize Lem's name, but do recognize "Stanislaw Lem" as a plausibly Polish name and make a logical guess

Players who are disadvantaged
- People who recognize clues about Lem and remember his name, but do not know that he wrote in Polish
- People who lose a buzzer-race on a giveaway that is decided on linguistic guessing

Players who are indifferent
- People who recognize clues about Lem and know that he wrote in Polish

From here, there's no necessarily "correct" answer as to whether Lem or Polish is the better answer choice - there are a series of advantages and drawbacks. However, I'm going to argue that Lem is a better answer here for several reasons:

1) Stipulating that some people will only convert this "Polish" tossup for linguistic reasons, I would argue such people are not demonstrating substantive engagement with the material enough to "deserve" points. I'll call this the "Japan Problem" as this crops up commonly in easy parts on Japan - bonus parts which give distinctively Japanese names and which have an answer of "Japan" may be testing legitimate academic material, but there will be some number of people who pick up the easy part perfunctorily ("find your ass" as it were) without knowing anything about the material.

2) Related to (1) - selecting a tossup which allows for linguistic conversion at the giveaway is more likely to produce buzzer races at the giveaway between weaker teams. This may be an idiosyncratic view of mine, but I think dead tossups are a fairer outcome (0-0) than what is essentially a coin flip / speed check for points.

3) Let's posit someone who only knows Stanislaw Lem's name, but not his origin, and has some degree of familiarity with Lem's work. In order for such a person to conclude that Lem wrote in Polish, they must be able to do one of the following:

a) Know that Stanislaw Lem is a Polish name AND assume that he wrote in Polish because of this (as opposed to maybe another language - after all, Polish people have emigrated to many places)
b) Infer some inherent connection to Poland/Polish identity from his work and also make the same assumption

Based on some very brief conversations I've had, it seems that a casual sci-fi reader would not necessarily be able to make inference (b) in the same way that, say, a reader of Nadine Gordimer's work would be able to infer she's South African, since Gordimer wrote a lot about apartheid. Thus, I think there are going to be some non-zero number of people who will have potentially read the books being asked about in this question. I think this is a real concern, and would guess that it's more likely that the converse scenario (oh, what's that Polish sci-fi author's name...) occurs when somebody has perhaps memorized Lem as "the Polish sci-fi author of Solaris" or whatnot as opposed to actually reading Lem's books.

CONCLUSION: On balance, it seems that using an answer of Polish is likely to help to muddy the battlefield for weaker teams by producing higher conversion rates for undesirable reasons, and is also more likely to punish some small, but non-zero number of the fraction of the audience that has interacted with Lem. Thus, I would argue that tossing up Polish instead of Stanislaw Lem is a poor choice in this scenario.

There are some other possible remedies besides changing the answer to Lem, of course:
- Include clues from other, easier Polish authors. This would produce more pre-giveaway buzzes and solve the "muddying the battlefield" problem, which I think is the biggest issue with tossing up "Polish," but it would reduce thematic coherence
- Use a different common-link to ask this material

I've gone way past my previous statement that I would not engage in set discussion in public fora anymore, so I should probably shut up at this point to avoid seeming like too much of a hypocrite. However, I would encourage other folks to engage in critical thinking of this sort about how people arrive at the correct answer to a tossup based on the clues. It looks really tedious to nitpick a single question like this, but I think this sort of analysis of "why am I choosing this answerline" can be broadly applied to answer selection in many other contexts and help us optimize the play experience of folks choosing to come to hard tournaments.

EDIT/ADDENDUM: I'd like to discuss a Poland tossup from a different tournament, in a different category, that I think implements the "include clues from easier authors late in the tossup" concept well:

A social scientist born in this country argued that individuals' "in-groups" and "out-groups" are a source of self-esteem as part of social identity theory. A social scientist from this country posited a "pointillist" conception of time in Consuming Life. This birthplace of Henri Tajfel was the birthplace of the University of Leeds professor who claimed the title event is neither a "Jewish problem" or a "German Problem" in Modernity and the Holocaust. A sociologist from this country claimed that humanity was not in an era of postmodernity but liquid modernity. In a 1922 book, another social scientist from this country examined the building of the waga and noted that necklaces passed clockwise while bracelets moved counter-clockwise in an analysis of the kula ring. For 10 points, name this home country of the author of Argonauts of the Western Pacific, Bronislaw Malinowski.

I think this tossup does a good job of testing hard material in a pyramidal fashion, while also giving weaker teams some pre-giveaway clues from a much better known thinker to buzz on before the question goes to clues that you can guess from linguistic knowledge, even if you know nothing about Malinowski (who is, needless to day, much more famous than Henri Tajfel and Zygmunt Bauman). You could make this into a two-thinker tossup on Zygmunt Bauman and Bronislaw Malinowski and I think it would work fine as well.

[Cheynem]

Can I see the original PIANO question on "Polish"?

[VSCOelasticity]

Can I see the original PIANO question on "Polish"?

11. ​A story in this language describes a ​Ulysses-like​ novel called “Gigamesh” whose syntactic structure forms
a blueprint of Notre Dame cathedral. Michael Kandel is best known for translating books from this language,
including a collection of nonexistent book reviews titled ​A Perfect Vacuum​. The heroes of a story in this
language defeat pirate Pugg by smothering him with printouts of all the information in the universe. A
collection in this language is divided into “sallies” like “The Mischief of King Balerion” and (*) ​“Trurl’s
Electronic Bard.” The ​Tales of Pirx the Pilot​ are in this language, which was also used for ​The Star Diaries​ of Ijon
Tichy. Two robotic “constructors” explore the universe in the ​Cyberiad,​ which was written in this language by an
author who also wrote about Kris Kelvin’s visit to a planet with a living ocean. For 10 points, name this language
used by the author of ​Solaris,​ Stanisław [​ sta-NEE-swoff]​ Lem.
ANSWER: ​Polish​ [or ​polski​] <DS>

[Cheynem]

If all of these clues are just on Lem, then the tossup (which isn't really a common link then, as Will points out) seems like it could have been better as a tossup just on Lem, unless you think Lem is too hard and that "Polish" is easier (which...I don't really buy).

Writing such a tossup, in which you are really referencing one answerline but opting to use a more unorthodox answerline, at least to me is done for the following reasons:

1. Trying to avoid transparency--for example, a tossup on the "Conservative Party" that only clues Margaret Thatcher is sometimes nice on easy difficulty levels.

2. Trying to be more creative on heavily mined topics--ergo, why tossups on the "green light," Gatsby's house, "New York," etc. are sometimes more appealing than another tossup on The Great Gatsby.

3. Trying to hyper-focus your tossup (a tossup on Citizen Kane only talking about opera, for example), which can also make a question slightly harder (do you actually know the state in which Goodbye, Columbus is set?), all of which can be helpful at higher difficulty levels.

I don't see any particular issue for 1, I don't think Lem is too heavily mined that we have to worry about 2 (and using "Polish" as the answerline doesn't do that), and I don't know about 3.

The easiest solution, I presume, would have been to make the tossup on Lem or make the tossup a true common link and find some other Polish science fiction text.

[CPiGuy]

2. Trying to be more creative on heavily mined topics--ergo, why tossups on the "green light," Gatsby's house, "New York," etc. are sometimes more appealing than another tossup on The Great Gatsby.

I think these questions are usually not only "more creative" but more difficult / demanding of more knowledge -- a tossup on the green light will not reward people who merely recognize characters or plot points from The Great Gatsby without knowing enough about the book to contextualize them.

[naan/steak-holding toll]

I've changed the title of this thread to reflect more accurately what the initial post was dissecting, since it's really not about common links. Thanks for the feedback, folks.

[DumbJaques]

I appreciate the analysis here, but I think these examples kind of illustrate the opposite point. To me, when you're tossing up works of literature, those are generally works that exist in a tradition - often most centrally, the tradition of the language in which its written. Comparatively, the fact that Tajfel or Malinowski were ethnically Polish is pretty unrelated to their work, or to its significance in anthropology. Despite having an aggressively Polish name, Malinowski spent all his professional time in Britain, the US, or the field. He influenced intellectual history in those countries, and didn't (to my knowledge) do any real work on Poland. I don't see much difference between asking that question and asking, say, this tossup from 2009 Science Non-Strosity:

One scientist from this present-day country has an eponymous large ring synthesis in which a dicarboxylic acid is converted to a cyclic ketone, which is used to synthesize animal musk. That scientist shared the 1939 Nobel Prize with Adolf Butenandt for partially synthesizing (*) testosterone. Another scientist from this country was the first to separate the enantiomers of Troger’s base, thus proving that nitrogen can be the chiral center of a molecule. That scientist is also the namesake of a priority rule along with Cahn and Ingold. In addition to Ruzicka and Prelog, another scientist from this country realized that seismic waves are refracted and thus must be passing through a boundary, while another scientist from this country worked for Westinghouse and is infamous for his “death ray”. For 10 points, name this homeland of Mohorovicic and Nikola Tesla, a former Yugoslav Republic with capital at Zagreb.
ANSWER: Croatia [accept Croats or Hrvatska] (3) [BA]

I suspect a lot of people would object to this today (and in fact, plenty of people objected to it then!). It's just not relevant to why you would care about these people. To use an apt phrase Clark Smith invoked when talking about some iffy clues in WORKSHOP, these facts are "trivially true," but not really substantive or pertinent.

I'm 100% in favor of raising answerability whenever and wherever you can. I just think the line should be drawn on moving away from contextual relevance, especially in tossups (this is less of an issue with a bonus, because you can cover it up with a later "also this is where -ski's come from" if someone doesn't know where Tajfel, a guy who was a French citizen and worked in England, happened to be born). This to me seems like a more important question than the rigorous analysis Will undertakes above - and one which is certainly easier for the question-writer to implement.

[Sam]

I suspect a lot of people would object to this today (and in fact, plenty of people objected to it then!). It's just not relevant to why you would care about these people. To use an apt phrase Clark Smith invoked when talking about some iffy clues in WORKSHOP, these facts are "trivially true," but not really substantive or pertinent.

"Trivially true" is a great phrase for clue criticism!
I really like a point Matt Weiner once made about question writing / answer line selection. You very rarely write a question on "George Washington" or "Stanislaw Lem" because you want to reward people who know a lot about George Washington or Stanislaw Lem. You want to reward people who know a lot about American history or 20th century literature. This is just a difference in mindset*: your clues are still mostly facts about Washington or Lem, but (for me, at least), it encourages a maybe more holistic approach to writing. And in the context of answer line selection, I think it encourages less "reification" of the answer line as an important determiner of difficulty or interesting-ness. The general squishy idea comes first, at least synchronically, and the answer line is just a means to an end.

This approach is also helpful in examining questions and trying to reverse engineer what field or subfield or sub-subfield's knowledge is being rewarded. In Will's second example, the clues are on a sociologist, psychologist, and anthropologist, the latter of whom was active before the first two were born. Would a person very knowledgeable about Polish intellectual thought have seen all of these? Possibly, though as Chris points out, these guys worked mostly outside of Poland. Would a sociologist have encountered all of them, or an anthropologist or psychologist? It doesn't seem like all three of them would be something that every major or even graduate student would encounter in course work, but maybe? The fact it seems difficult to say what knowledge is being rewarded seems worrying, though I also don't know enough about them to say for sure. This is just repeating Chris's point, of course.

(I also think Conor is correct that Mike Cheyne's second point is usually the same as his third one. Especially for well-combed topics, there are often already a bunch of lists or SparkNotes on characters or specific symbols. Things like "Gatsby's house" are usually memorable to people who read the book but don't necessarily show up in these kinds of summaries.)

* It's also obviously very subjective what the larger field your question falls in is. Do you want to reward people who know "American history" or "18th century American history?" Or "American military history?" Or "Atlantic history?" Any of those are fine. The important thing is it's not "people who have recently read this Britannica article on Washington," even if that's where most of the clues come from.