Variable question difficulty

Dormant threads from the high school sections are preserved here.
Locked
STPickrell
Auron
Posts: 1350
Joined: Fri Nov 28, 2003 11:12 pm
Location: Vienna, VA
Contact:

Variable question difficulty

Post by STPickrell »

Greetings all:

The biggest complaint I have received on VHSL series questions has been the variable difficulty between rounds. I've noticed some of the rounds my team and I write seem to get 350+ points a game between two decent teams, whereas other rounds seem to have trouble with those same two teams getting 250 points a game. I would like to have the questions so that two terrible teams would have a hard time getting under 200 combined a game.

How do the other question writers deal with this difficulty? My difficulty rating system where I hired a few smart folks to help me rate the questions will help somewhat, I think, but what other methods do people have to end this difficulty?

Thanks for any advice people might give.
Shawn Pickrell, HSAPQ CFO
David Riley
Auron
Posts: 1401
Joined: Tue Apr 29, 2003 8:27 am
Location: Morton Grove, IL

Post by David Riley »

I think question difficulty (as opposed to questions that are too easy) is difficult to gauge [sic] if you are providing rounds on a national scale. For example, in Illnois, we are allowed 30 seconds for computation questions; thus, calculus questions tend to be the norm here [at least, north of I-72]. I've written questions in the past that I have thought would be within the high school curriculum, only to have students stare blank-faced when they heard them. For example, my seniors last year did not know that Tolstoy wrote War and Peace [for which they were severely chastised]. This may sound like an extreme example, but I've heard worse.

Generally, if the strong teams have a point spread that is less than half the total points, then I believe the questions may be too difficult.
User avatar
First Chairman
Auron
Posts: 3651
Joined: Sat Apr 19, 2003 8:21 pm
Location: Fairfax VA
Contact:

Post by First Chairman »

The toughest part is trying to impose a difficulty standard on an activity in which our questions are not "standard". We're not like standardized tests where there is a different strategy. This is about recall, and it's tough to predict what kids know. Of course, we are in the business to also confirm what kids ought to know or may be interested to know. How that all fits I guess depends on the audience you write for, but a lot of it is about knowing the audience too. Everything is so different from area to area that it is tough to figure out.
Emil Thomas Chuck, Ph.D.
Founder, PACE
Facebook junkie and unofficial advisor to aspiring health professionals in quiz bowl
---
Pimping Green Tea Ginger Ale (Canada Dry)
bdavery
Lulu
Posts: 95
Joined: Mon Aug 04, 2003 11:29 am
Location: Centennial CO
Contact:

question difficulty

Post by bdavery »

One way to help with difficulty is to put one subject at a time into a tournament. For example, if you're doing 10 rounds and each has to have a US history question, then rounds 1-3 might include George Washington and Bill Clinton as answers, while rounds 8-10 might have Battle of Chickamauga and the Specie Circular.

No matter what you do, someone will be unhappy--and as Dave points out, if your players don't know who wrote War and Peace (which would likely be a round 1-3 answer), there's not a whole lot a writer can do.
Bryce Avery
President, Avery Enterprises
http://www.averyenterprises.net/blog
STPickrell
Auron
Posts: 1350
Joined: Fri Nov 28, 2003 11:12 pm
Location: Vienna, VA
Contact:

Post by STPickrell »

Well, I guess my system (I'll call it the Wise Men System, Districts will get 4 Wise Men and Regions/States will get all 7) is the best I can do.

I tend to get a sinking feeling when I hear of matches where the VHSL scores are under 250 combined (50% of the questions). So far only one district that I know of has achieved that distinction for an average score.

GW and Bill Clinton, I'd give a "1" to. Battle of Chickamauga I'd give a "3" to on a 1-7 scale. Specie Circular gets a "5". JMO ... :-)
Shawn Pickrell, HSAPQ CFO
leapfrog314
Wakka
Posts: 248
Joined: Mon Sep 12, 2005 10:49 am
Contact:

Post by leapfrog314 »

The problem is mostly alleviated if questions are not too short, and are pyramidal enough. You can write hard questions if they have obvious last clues; they don't degrade into buzzer races because the player with more knowledge would have buzzed in earlier. And if it takes until the giveaway, well, points are still being scored.

Also, as a player, the biggest problem with varying question difficulty too much is that the player doesn't know when to buzz in. If the questions are all easy and short, they are comfortable buzzing in early, because they know their initial thought is likely to be correct. On the other hand, if some of the questions are hard, then nobody knows when to buzz in. By the way, your questions at Fremd were definitely not the worst that I've played in that regard. The worst is when questions alternate between four sentences and one sentence. Ack!
STPickrell
Auron
Posts: 1350
Joined: Fri Nov 28, 2003 11:12 pm
Location: Vienna, VA
Contact:

Post by STPickrell »

Another factor to consider is that there is a directed round, of 10 questions given to each team, with the potential for bounceback if the initial team can't get it. So for Regionals I have tried to get the average difficulty and standard deviation of difficulty (as determined by the Wise Men) to be the same between Set A and Set B in the match.
Shawn Pickrell, HSAPQ CFO
User avatar
Stained Diviner
Auron
Posts: 5085
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland
Contact:

Post by Stained Diviner »

Regional differences are significant. In Illinois, we have lots of math, so teams recruit math specialists and gets lots of practice in it. We have very little legitimate current events, so good teams sometimes don't know about major stories.

Also, around here Specie Circular would be considered easier than Chickamauga. We don't get to take class field trips to Civil War battle sites and generally have less interest in the Civil War than Virginians do, especially Confederate victories. (BTW, I do know that Chickamauga is not in Virginia.)

Also, sometimes questions that are designed to be super easy end up getting missed. If you wanted to write a real easy American Lit tossup, you could look up best-selling books of all time, and ask who wrote books such as Gone With The Wind or Roots. However, those books are not currently read in a lot of high schools, and teams that only go to one or two tournaments a year and do not study for meets may not know the answer. Some basic facts of literary history are not completely common knowledge.
David Reinstein
Head Writer and Editor for Scobol Solo, Masonics, and IESA; TD for Scobol Solo and Reinstein Varsity; IHSSBCA Board Member; IHSSBCA Chair (2004-2014); PACE President (2016-2018)
MahoningQuizBowler
Rikku
Posts: 417
Joined: Fri Jan 23, 2004 10:58 am
Location: Youngstown, Ohio
Contact:

Post by MahoningQuizBowler »

My two cents from experiences in Youngstown...

In the Ohio format, social studies-based categories make up a large portion of the game (World and American History, Government, and Geography) with math/science next (Math, Life and Physical Sci) and art/literature bringing up the rear (American and World/British Lit, Fine Arts).

Even though the points are the same, teams place (generally) much less emphasis on literature than sciences, and do worse in both areas combined than social studies. However, the teams that commit to working on those areas win. In each of the three seasons I've kept category statistics for the Mahoning County league, the team that has won the literature categories has won the league. Since American Lit opens an Ohio match, the ability to jump out to an early 6-0 or 7-0 lead is key to taking control of the match and keeping it.

This season, what we've been doing is adding more of the 3, 4, and 5 Pickrell scale questions into the traditionally strong categories, and more 1 and 2 questions into the less answered. It's worked so far...all of the teams who lead the categories are averaging about 3 points a round in each category (max 7).
User avatar
Stained Diviner
Auron
Posts: 5085
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland
Contact:

Post by Stained Diviner »

Mr. Avery makes the point above that early rounds should have easy questions and later rounds should have difficult questions. My opinion is that this change is often overdone. If too many very easy questions are put in an early round, then a knowledgeable team can be upset by a team that gets on a roll answering questions that everybody knows the answer to. Also, if the questions in the last round or two are too difficult, then even the good teams will have low scores and the match will come down to one or two questions.

There should be some combination of difficult and easy questions in each round. The issue of students not knowing when to buzz in can be handled by making very clear lead-ins.

This is just my opinion; I think that reasonable people can disagree on this one.
David Reinstein
Head Writer and Editor for Scobol Solo, Masonics, and IESA; TD for Scobol Solo and Reinstein Varsity; IHSSBCA Board Member; IHSSBCA Chair (2004-2014); PACE President (2016-2018)
User avatar
First Chairman
Auron
Posts: 3651
Joined: Sat Apr 19, 2003 8:21 pm
Location: Fairfax VA
Contact:

Post by First Chairman »

There are many arguments for this, since I sometimes ascribe to at least a version of the ideas I'm going to state below.

In general, there should be as little difference in difficulty among prelim rounds. For question difficulty, it should not matter whether a significantly good team plays a significantly not-so-good team in the first prelim round or the last prelim round. I like scheduling a rivalry game between the "best two teams" as the last game in a prelim schedule as much as I can, but you still should treat the question-writing to be as unbiased towards that result as possible (as if all matches were randomly drawn after every round).

As for whether there should be an increase in difficulty for playoffs, I am fairly torn by it. I think there is an expectation for an increase in difficulty but only to try to distinguish the better of two really good teams based on depth of knowledge and buzzer speed (note my order of things is my opinion). You change the element of speed by having different question styles (one-liners promote faster speeds than pyramidals), but you change depth of knowledge with the content of the questions. No, I don't recommend changing question styles in a game... that would be too disruptive to game flow. (Playing tennis... move the net... send in the llamas.)

That said (and I know I haven't always ascribed to this), despite most people using single elimination as their playoff system, difficulty really should not vary significantly among playoff packets. We claim we write a "finals packet" or a "semifinals packet"... that's solely because of when those questions come up. They should really not be written as entities of their own accord. That's where I think you get these wild raises in difficulty. Some questions perhaps do belong in a finals game, but don't make your finals game up of such questions exclusively. You can have your "championship set of words" like they do at the National Spelling Bee, or your championship set of questions with the Geography Bee... but they should be able to distinguish one very good team from another, not determine whether the teams are good in the first place.
Emil Thomas Chuck, Ph.D.
Founder, PACE
Facebook junkie and unofficial advisor to aspiring health professionals in quiz bowl
---
Pimping Green Tea Ginger Ale (Canada Dry)
Locked