Judging difficulty.

Dormant threads from the high school sections are preserved here.
Locked
User avatar
at your pleasure
Auron
Posts: 1670
Joined: Sun Aug 03, 2008 7:56 pm

Judging difficulty.

Post by at your pleasure »

I have been thinking about the tournaments so far this season, and noticed that one of the most common issues was tournaments not hitting their target difficulty(. Most of the time, difficulty was overshot. With that in mind, I would like to pose the following question: How can we, as writers and editors, do a better job hitting a tournament's "target difficulty"?
A couple of ideas I'll throw out:
-Set up a searchable database of high school questions on the same lines as Gyankoosh so people can quickly check whether their answer has come up before.
-Produce more avenues for canon expansion so we don't wind up doing it by writing overly difficult hard bonus parts. By "more avenues" I have in mind subject tournaments and explicitly higher-level events.
Any other ideas?
Douglas Graebner, Walt Whitman HS 10, Uchicago 14
"... imagination acts upon man as really as does gravitation, and may kill him as certainly as a dose of prussic acid."-Sir James Frazer,The Golden Bough

http://avorticistking.wordpress.com/

User avatar
Matt Weiner
Sin
Posts: 8413
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Judging difficulty.

Post by Matt Weiner »

Anti-Climacus wrote:-Set up a searchable database of high school questions on the same lines as Gyankoosh so people can quickly check whether their answer has come up before.
-Produce more avenues for canon expansion so we don't wind up doing it by writing overly difficult hard bonus parts. By "more avenues" I have in mind subject tournaments and explicitly higher-level events.
Any other ideas?
I think these are dangerous models, since they rely on the idea of the serious quizbowl player who is more likely to know things because they have come up before. Those players are perfectly capable of answering questions on "Teach Us to Outgrow Our Madness"; it's the average player who is not, and who difficulty becomes a concern for. So, I'd caution against any standard that ignores absolute difficulty in favor of lowering difficulty precisely for the people who don't really need low-difficulty questions.
Matt Weiner
Founder of hsquizbowl.org

User avatar
Deviant Insider
Auron
Posts: 4672
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland
Contact:

Re: Judging difficulty.

Post by Deviant Insider »

I thought that Gyankosh had a high school section, though the site seems to be down right now.

Also, I agree with Matt that methods of expanding the high school canon only work for the elite teams, and they don't really do much for many of the elite teams since those teams are already looking at college questions anyways.

If you want to write/edit a high school tournament that has a decent conversion rate for teams that are not among the top hundred in the country, then you have to choose your answer space carefully. You can use HSAPQ as a guide, keeping in mind that even though those questions sound easy to the top teams, they are at the upper level of average high school difficulty. If your questions are more difficult than that, even by a little bit, then you should tell teams before they register that your tournament is using challenging questions.

As an example, there has been an increase in the past year or two in the number of questions on Chinese and Japanese literature. However, if there was a match between two teams that were not in the top 1000 in the country (out of 3000+) and there was a Japanese Lit question, the only answer guessed would be Tale of Genji, even if you asked for a 20th Century author. That doesn't mean that you should never toss up Oe, but it does mean that such a question should be considered difficult and probably should be balanced out by having some much easier lit tossups in the same round, unless you are gearing your tournament towards strong teams.

Writing in a small answer space may not be fun for some writers, and it requires a lot of effort to avoid transparency, but that's how it goes.
David Reinstein
PACE VP of Outreach, Head Writer and Editor for Scobol Solo and Masonics (Illinois), TD for New Trier Scobol Solo and New Trier Varsity, Writer for NAQT (2011-2017), IHSSBCA Board Member, IHSSBCA Chair (2004-2014), PACE Member, PACE President (2016-2018), New Trier Coach (1994-2011)

User avatar
Down and out in Quintana Roo
Auron
Posts: 2907
Joined: Wed Apr 09, 2008 7:25 am
Location: Camden, DE
Contact:

Re: Judging difficulty.

Post by Down and out in Quintana Roo »

Shcool wrote:If you want to write/edit a high school tournament that has a decent conversion rate for teams that are not among the top hundred in the country, then you have to choose your answer space carefully. You can use HSAPQ as a guide, keeping in mind that even though those questions sound easy to the top teams, they are at the upper level of average high school difficulty. If your questions are more difficult than that, even by a little bit, then you should tell teams before they register that your tournament is using challenging questions.
This.

I've found that the absolute best questions being produced right now are by HSAPQ, so for more reasons than one, they are the ones to which you should refer to get the proper difficulty scale for an "average" tournament if you know you're going to try to attract some at least pseudo-top-level teams.
Mr. Andrew Chrzanowski
Caesar Rodney High School
Camden, Delaware
CRHS '97-'01
University of Delaware '01-'05
CRHS quizbowl coach '06-'12
http://crquizbowl.edublogs.org

User avatar
Terrible Shorts Depot
Yuna
Posts: 760
Joined: Mon Mar 24, 2008 5:05 pm
Contact:

Re: Judging difficulty.

Post by Terrible Shorts Depot »

High school difficulty is a wildly different beast than college difficulty. Due to the wide variation in high school "quality"*, the "guaranteed" knowledge base is actually rather small, whereas, in college, I would contend that this "guaranteed" base is much wider, deeper, and more varied. This poses a great problem for HSAPQ, NAQT, and all other national question vendors. Some answers (say, a random Japanese author) regarded as simple in DC are most likely considered to be quite difficult in Oklahoma. In reference to the difficulty of sets, most every house write I've seen has been too hard for most of the nation. HSAPQ seems to hit what I would call "good" difficulty very well. NAQT flucuates. So, as a result, I would advocate, at least for "national" sets, a rather low difficulty. It really is okay if State College or Charter puts up 27 points per bonus. If one is house writing a set, difficulty could probably be raised a bit, because, in this case, the authors know the audience that they are writing for.


*Quality is short hand for something approximating "rigor of school, socioeconomic standing of pupils, dedication of quiz bowl team, etc."
Charlie Rosenthal
Shady Side Academy '09
Carleton College '13
University of Pennsylvania '18

Adventure Temple Trail
Auron
Posts: 2617
Joined: Tue Jul 15, 2008 9:52 pm

Re: Judging difficulty.

Post by Adventure Temple Trail »

Charlie and Mr. Reinstein pretty much said everything I had to say pretty well, but here are a few additional thoughts:

-The best way to control difficulty is to have a target difficulty. If you want to write a tournament with 5-line questions and IS difficulty, which is perfectly fine by me, it's important to aim for (and edit to) that difficulty. The same goes for 7-line nationals-difficulty tournaments. Moreover, it's probably better to announce target difficulty up-front so people have the proper comparison point: part of the great reception for Prison Bowl was that it hit the expected slightly-above-averag-e difficulty target right on the dot.

-Ultimately, it is important to depersonalize and consider all facets of an audience. The "ok i know this so its stock" attitude is a trap waiting for many writers - it's important to challenge the area's top teams with the leadins &third parts, but the giveaways and pre-giveaway clues should be accessible to the middling/bottom teams. I personally believe that "For 10 points, name this element with symbol H" is a bit easy, but it might not even be - it all depends on the audience.

-As Mr. Reinstein notes, there are certainly answers that have come up in quizbowl more frequently recently than before, and answers that have come up more frequently in high school quizbowl than in high school classrooms. While Oe is fine in a regular set, it's important for us to be aware that he is on the more difficult end of the acceptable canon, especially for newer and less experienced teams whose knowledge base starts with things they learn outside of quizbowl itself. While one important element of quizbowl involves exposing teams and players to things they might not have otherwise heard of - for the sake of scoring more points and for the sake of learning-in-itself - it's important to balance "expanded canon" answers, like Oe, even, with accessible ones.

On a sidenote: Can we define what "regular difficulty" even is for high schoolers? It would be cool to see some input from HSAPQ or NAQT insiders, or experienced housewriters, about what they try to achieve with balancing out difficulty, percentage of teams that should get the average tossup, etc.
Matt J.
ex-Georgetown Day HS, ex-Yale
member emeritus, ACF

Try my original crossword puzzles

User avatar
First Chairman
Auron
Posts: 3875
Joined: Sat Apr 19, 2003 8:21 pm
Location: Fairfax VA
Contact:

Re: Judging difficulty.

Post by First Chairman »

Matt hits a concept regarding answer space that is very important: ultimately most of your answers have to be recognizable by the middle 50% of the players in the field for a regular tournament. No student is going to ALWAYS want to learn about something he/she hasn't heard of unless there is an obvious connection.
Emil Thomas Chuck, Ph.D.
Founder, PACE
Facebook junkie and unofficial advisor to aspiring health professionals in quiz bowl
---
Pimping Green Tea Ginger Ale (Canada Dry)

Locked