Advice on question writing companies

Dormant threads from the high school sections are preserved here.
Locked
User avatar
goppel
Kimahri
Posts: 5
Joined: Sat Dec 13, 2003 2:41 am
Location: Edmond Memorial - Edmond, Oklahoma
Contact:

Advice on question writing companies

Post by goppel » Thu Dec 18, 2008 1:13 am

Our state is bidding on a new vendor for our tournament questions. Assuming this is an appropriate forum, can you share your opinion of Avery Enterprises and Questions Gallore?
thanks!
Edmond Academic Team "Cave Canem"
Edmond Memorial High School - Edmond, OK

User avatar
Matt Weiner
Sin
Posts: 8411
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Advice on question writing companies

Post by Matt Weiner » Thu Dec 18, 2008 1:17 am

People certainly can and will do that. However, be aware that you asked a question and will get a brutally honest answer.
Matt Weiner
Founder of hsquizbowl.org

User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5640
Joined: Wed Jul 26, 2006 11:46 pm
Location: Columbia, MO

Re: Advice on question writing companies

Post by Sen. Estes Kefauver (D-TN) » Thu Dec 18, 2008 1:26 am

Selecting either of those companies will result in a set that is nothing short of an insult to students who enjoy playing this game.
Bryce Avery has a track record of ignoring criticism, threatening lawsuits, and name-calling at people who are young enough to be his child, as well as producing questions that are riddled with factual errors and buzzer races.
Questions Galore, while not coming with the ethical baggage that Bryce Avery brings to the table (as far as we know), produces sets that are even worse in quality, have impossible calculations, multiple questions per round on things nobody in high school can reasonably be expected to know, a lack of giveaway clues that might actually help conversion, nothing but speed checks for the questions that are answerable, and a propensity for blatantly ignoring the distribution they are assigned.
If I were you, I would steer clear of either of these options.
Last edited by Sen. Estes Kefauver (D-TN) on Thu Dec 18, 2008 1:00 pm, edited 1 time in total.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White

User avatar
cvdwightw
Auron
Posts: 3446
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Advice on question writing companies

Post by cvdwightw » Thu Dec 18, 2008 2:50 am

It is the general consensus of many on this board, supported by varying degrees of hard evidence, that coaches of bad teams will generally attempt to get short, buzzer-race questions often rife with factual errors, because the deciding factor is then buzzer speed and not knowledge, and thus they have a chance to spring the upset on even the most knowledgeable teams. This can be incredibly frustrating to players who are either on good teams or are willing to put in the effort to become good.

Any company that cannot offer pyramidal questions which have difficult information at the beginning and taper down to an easily accessible giveaway should, plain and simple, not even be considered. The reason for this is that non-pyramidal questions must use the relative obscurity of answers to differentiate between teams of different knowledge levels, so more questions go dead in more matches. Thus, these questions are generally not actually as fun to play as questions where most teams will know the answer, even if players do have to wait for the giveaway clue to answer the question.

Based on what little I have seen from these companies, neither of them can make an honest claim to provide pyramidal questions. However, as I have not had direct interactions with either of these companies, I would check with third parties to verify whether or not their questions are pyramidal.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry

User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5640
Joined: Wed Jul 26, 2006 11:46 pm
Location: Columbia, MO

Re: Advice on question writing companies

Post by Sen. Estes Kefauver (D-TN) » Thu Dec 18, 2008 3:00 am

You can check my above post as one of those third parties - I've played 6 tournaments on Avery and 1 on Questions Galore (that was heavily edited because there was almost no literature in the original set) as well as seen many of their sets in practice and moderated at one tournament held on them. All of these providers are unacceptable and not worthy of the money that your state would be paying them.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White

David Riley
Auron
Posts: 1428
Joined: Tue Apr 29, 2003 8:27 am
Location: Morton Grove, IL

Re: Advice on question writing companies

Post by David Riley » Thu Dec 18, 2008 8:46 am

Ditto to everything Charlie said. I've coached teams who have played at tournaments using these questions, and I wouldn't recommend either of them. :bees:

You might try writing letters (better yet, get coaches and students of strong or want-to-be strong teams to write letters) to the members of the Oklahoma board recommending a better vendor, with some examples and samples. In other words, be part of the solution.

I know many people on this board do not like NAQT A Level sets, but that would be a start and hundreds of times better.
David Riley
Coach Emeritus, Loyola Academy, Wilmette, Illinois, 1993-2010
Steering Committee, IHSSBCA, 1996 -
Member, PACE, 2012 -

"This is 1183, of course we're barbarians" -- Eleanor of Aquitaine in "The Lion in Winter"

Kanga-Rat Murder Society
Wakka
Posts: 209
Joined: Fri Mar 07, 2008 10:52 pm
Location: NW Suburbia, IL

Re: Advice on question writing companies

Post by Kanga-Rat Murder Society » Thu Dec 18, 2008 9:42 am

Our conference uses Avery Questions and I cannot tell you how horrible they are. They are not pyramidal in the sense that the first clue is often the easiest clue. All literature seems to be pop lit, meaning you will hear more questions on Harry Potter than you will on the works of Tolstoy or Dickens. The math always seems to be really easy or really hard. We recently had the tossup "Multiply (1/10) times (2/3)." The best advice that I can give you is to not use them. I have not played Questions Galore enough to tell you my opinion, but I know that they are not well thought of. My advice would be to go with other companies, even if it costs you more.
Nicholas Bergeon
Buffalo Grove High School '09
UW-Madison '12
WUSTL Law'15

STPickrell
Auron
Posts: 1501
Joined: Fri Nov 28, 2003 11:12 pm
Location: Vienna, VA
Contact:

Re: Advice on question writing companies

Post by STPickrell » Thu Dec 18, 2008 10:27 am

I know the OSSAA wants questions to be two lines or shorter. Sadly, many of the pyramidal companies will not write questions that short, as (and I agree with them) those are simply too short to adequately distinguish between the knowledge levels of the two teams.

I would strenuously advise the OSSAA to consider letting questions at regions and states be 3-4 lines due to the higher talent levels of the teams involved.

If the format cannot be changed, here is what I would do:
1. Seek a question writer that will *attempt* to put 2-3 clues in a question. e.g. "Who created the character of A in book B and the character of C in book D?" as opposed to "Who wrote book D?" or even worse, questions with 1.5 lines of fluff "He wrote novels full of suspense and drama. Who wrote book D?" or a 90-degree turn "Someone else wrote book A. But who wrote book D?" or questions that ask "why?" or "how?" (Unless, of course, 10-minute protests or asking the teacher who's volunteered to help out are your cup of tea.)
2. Someone that at least attempts to follow a distribution, and doesn't let pet categories dominate within said distribution.
3. Someone that is factually correct, and includes alternate acceptable answers/answers that are close and shouldn't be prompted).

As for lightning rounds, you want someone that composes lightning rounds that are (1) answerable and (2) written so they can be completed in 60 seconds. Is there a reason the time cannot be extended to 90 or 120 seconds? This allows for multi-discipline lightning rounds which are, IMO, better (e.g. 'things that start with B' or 'things related to the month of April.')

Is what I advise the ideal? Of course not.

Is it an improvement? I believe it is. I believe (and others might disagree) that it is better to play quizbowl that is 20% of the ideal instead of quizbowl that is 10% of the ideal.

Good luck. Are there any other bidders?
Shawn Pickrell, HSAPQ CFO

User avatar
rjaguar3
Rikku
Posts: 278
Joined: Tue Dec 05, 2006 8:39 am

Re: Advice on question writing companies

Post by rjaguar3 » Thu Dec 18, 2008 10:31 am

BG MSL Champs wrote:We recently had the tossup "Multiply (1/10) times (2/3)."
That's Numberwang!

Unfortunately, I know that I played on Avery. As far as I remember (this was 2005-06 DVC conference), the questions lacked any semblance of pyramidality. I don't remember any skewed canon, but, then again, nearly EVERY tournament that year had 1/1 Harry Potter (or so it seemed).

As for QG, our school's F/S tournament we run uses these questions. I've agreed to edit them, so I'll report once the tournament is held just how bad the questions were. In the meantime, you can view some of last year's questions here.
Greg (Vanderbilt 2012, Wheaton North 2008)

User avatar
First Chairman
Auron
Posts: 3875
Joined: Sat Apr 19, 2003 8:21 pm
Location: Fairfax VA
Contact:

Re: Advice on question writing companies

Post by First Chairman » Thu Dec 18, 2008 11:42 am

All my opinion.

Just wondering: can you post the call for proposals? From my own standpoint, I would like to find out what Oklahoma is specifically looking for and values in the questions to pick the vendor that best fits its programmatic goals. How are the vendors going to be accountable for satisfying those criteria? I say this because as it has been pointed out, if the selection committee is being told by its membership to find writers who can produce a bunch of original one-sentence buzzer-race questions with 40% math calculation without justifying how that distribution meets the programmatic goals of Oklahoma (or other state) quiz bowl, then there are going to be problems. I presume each of those vendors sent in sample packets from previously used events, and a sampling of positive references?

Yes, squeaky wheels that many people are describing as "bad teams" are going to be more vocal than the "good teams" because in competitions like this, there will be more teams that do not win the state championship (or even advance) than there are teams who will win. I agree that everyone in education would like to see more parity in a competitive activity, but I also note that there are other ways to do this than lowering curricular standards. How amenable quiz bowl (state, nationally) is to pursuing these other avenues is another discussion.

But yes, the advice you receive here from many of the better teams and coaches is very valuable as you evaluate your proposals, and I would always want to know with any q-writing company appropriate probative questions on how they adjust appropriately to feedback.

The other major thing is question delivery. That is always an issue that I think many complain about (will you deliver questions when I ask you to... say two weeks before the tournament).

The last is appropriate proprietary issues: are you going to summarily edit questions from your contractor without the contractor's permission, then claim any problems with the questions are the fault of your contractor? I think that if this continues, I would expect to see q-writing companies have a stated warranty on their questions where users will be subject to an additional fee if any of the question sets are modified without prior permission. That's like buying a Mercedes, but then deciding to put in your own engine. How far should q-writing companies go to customize their question sets to conform to certain game formats should be explicit, and purchasers should be upfront with their intentions on why they are looking to work with said company. Furthermore, if you are going to take that much time to edit someone else's work, you might as well write your own.

Since I haven't purchased questions in ages for tournaments (from vendors), I'm just relying on what people have posted here. I don't think there is such acrimony with a lot of the companies that are being used most frequently, but I think that the quality of our competitions can be much improved if appropriate communication of expectations and subsequent/frequent progress and follow-through is done. To that end, I'm not sure that most of our cottage q-writing companies have reached that level of customer service (nor should I expect them to) so I think anything that can be done to strengthen them through constructive feedback would be great.

Shawn's post is one example of articulated question expectations that I guess I'd like to see each vendor state philosophically and each purchaser articulate. I think Illinois and Kentucky do this clearly, and whether people like that game format or not, that's up to each state association/host to decide.
Emil Thomas Chuck, Ph.D.
Founder, PACE
Facebook junkie and unofficial advisor to aspiring health professionals in quiz bowl
---
Pimping Green Tea Ginger Ale (Canada Dry)

User avatar
Deviant Insider
Auron
Posts: 4565
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland

Re: Advice on question writing companies

Post by Deviant Insider » Thu Dec 18, 2008 12:39 pm

For the IHSA (Illinois) State Series, the questions are written and edited by a fairly large group of former coaches and players, some of whom have no connection to Illinois. This approach, which came about after years of being frustrated by a particular vendor who is not one of the two you mentioned, has its pros and cons. The cons are that it ends up being a lot of work, since willing writers are sometimes difficult to find, writers sometimes back out of commitments, and it is pretty much impossible to get college students and former coaches to agree on what constitutes good questions. The pros are that we can put somebody in charge who makes sure that the questions meet our constraints and that deadlines are met. Additionally, we have gotten many very good writers who would not have been interested in writing the entire State Series but have been willing to contribute a fair number of very good questions.

The questions we end up with are certainly imperfect, though I believe that they are better than what is available from commercial vendors willing to write our State Series in our format.

I am sure that BA and QG submitted plenty of sample questions to you, and they both have a long history of delivering lots of questions on time to lots of tournaments. At some point, this decision will be made by the people in charge looking at the samples and deciding what they like better. However, much of the advice given above is very good advice. In Quizbowl, Issue #1 is Question Quality. A well-run Quizbowl organization with bad questions is like a well-run sports franchise with bad athletes. Take a look at different questions that are out there, including ones found here, here, and here, and ask yourself which questions do the best job of promoting learning. Because Quizbowl is largely about learning, those probably are the best questions.
David Reinstein
Head Writer and Editor for Scobol Solo and Masonics (Illinois), TD for New Trier Scobol Solo and New Trier Varsity, Writer for NAQT (2011-2017), IHSSBCA Board Member, IHSSBCA Chair (2004-2014), PACE Member, PACE President (2016-2018), New Trier Coach (1994-2011)

User avatar
at your pleasure
Auron
Posts: 1670
Joined: Sun Aug 03, 2008 7:56 pm

Re: Advice on question writing companies

Post by at your pleasure » Thu Dec 18, 2008 5:22 pm

Good vendors:
-NAQT. While shorter and flawed, the questions are still pyramidal. IS sets are varsity sets, A sets are junior varsity sets. A sets are terrible, and IS sets have serious problems, but they are a major improvement over the vendors you mentioned. I belive the NAQT website has a sample packet.
HSPAQ-probably the best vendor whose sets I've played. Good questions, good distribution. There's a link to a sample packet in the discussion forum.
Ageis-I've never played their sets, but they have a good reputation, so you should probably check them out.
Douglas Graebner, Walt Whitman HS 10, Uchicago 14
"... imagination acts upon man as really as does gravitation, and may kill him as certainly as a dose of prussic acid."-Sir James Frazer,The Golden Bough

http://avorticistking.wordpress.com/

User avatar
Terrible Shorts Depot
Yuna
Posts: 760
Joined: Mon Mar 24, 2008 5:05 pm
Contact:

Re: Advice on question writing companies

Post by Terrible Shorts Depot » Thu Dec 18, 2008 7:19 pm

Question: how much computational math does Oklahoma want? This will greatly influence who will be willing to write.
Charlie Rosenthal
Shady Side Academy '09
Carleton College '13
University of Pennsylvania '18

User avatar
at your pleasure
Auron
Posts: 1670
Joined: Sun Aug 03, 2008 7:56 pm

Re: Advice on question writing companies

Post by at your pleasure » Thu Dec 18, 2008 9:09 pm

Well, if Oklahoma coaches wil revolt at the suggestion that mathcomp should be kept to a minimum or even eliminated, I think NAQT would allow hosts to add mathcomp.
Douglas Graebner, Walt Whitman HS 10, Uchicago 14
"... imagination acts upon man as really as does gravitation, and may kill him as certainly as a dose of prussic acid."-Sir James Frazer,The Golden Bough

http://avorticistking.wordpress.com/

User avatar
Terrible Shorts Depot
Yuna
Posts: 760
Joined: Mon Mar 24, 2008 5:05 pm
Contact:

Re: Advice on question writing companies

Post by Terrible Shorts Depot » Thu Dec 18, 2008 9:21 pm

Anti-Climacus wrote:Well, if Oklahoma coaches wil revolt at the suggestion that mathcomp should be kept to a minimum or even eliminated, I think NAQT would allow hosts to add mathcomp.
NAQT IS and IS-A sets have a pretty good amount of mathcomp. Also, I believe that NAQT creates special sets for customers with specific requests. I'm sure NAQT would be willing to up their mathcomp to get the Oklahoma contract. Hell, if Oklahoma wanted, they could have NAQT speedcheck sets, which would be, umm, bad for everyone.
Charlie Rosenthal
Shady Side Academy '09
Carleton College '13
University of Pennsylvania '18

cdcarter
Yuna
Posts: 945
Joined: Thu Nov 15, 2007 12:06 am
Location: Minneapolis, MN
Contact:

Re: Advice on question writing companies

Post by cdcarter » Thu Dec 18, 2008 9:31 pm

la2pgh wrote:
Anti-Climacus wrote:Well, if Oklahoma coaches wil revolt at the suggestion that mathcomp should be kept to a minimum or even eliminated, I think NAQT would allow hosts to add mathcomp.
NAQT IS and IS-A sets have a pretty good amount of mathcomp. Also, I believe that NAQT creates special sets for customers with specific requests. I'm sure NAQT would be willing to up their mathcomp to get the Oklahoma contract. Hell, if Oklahoma wanted, they could have NAQT speedcheck sets, which would be, umm, bad for everyone.
Not being from NAQT, I can't officially comment on this, but I know NAQT has some special policies about doing state series sets that probably don't align with the Oklahoma bid.
Christian Carter
Minneapolis South High School '09 | Emerson College '13
PACE Member (retired)

User avatar
goppel
Kimahri
Posts: 5
Joined: Sat Dec 13, 2003 2:41 am
Location: Edmond Memorial - Edmond, Oklahoma
Contact:

WOW

Post by goppel » Thu Dec 18, 2008 10:41 pm

I was warned that my question would elicit some frank answers and I guess I got them.

Without violating the ethical obligations of my role as a member of the OSSAA Advisory Board, let me just say that my question was based on the fact that the aforementioned vendors made bids and were unknown to me.

I appreciate the very specific and pointed comments. We can probably close this forum because the bidding process is closed and most of the suggestions are irrelevant. Is there a thread here that discusses vendors in general? That might help serve the general purpose of what I was trying to do here.

Thanks for your help.
Edmond Academic Team "Cave Canem"
Edmond Memorial High School - Edmond, OK

User avatar
Deviant Insider
Auron
Posts: 4565
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland

Re: Advice on question writing companies

Post by Deviant Insider » Fri Dec 19, 2008 1:38 am

You might want to look through these links.

I understand that many of the comments above probably are too late to be relevant to this year, but I think many of them should be relevant if you are going to take a long-term view of what you want Oklahoma Quizbowl to be like in five or ten years.
David Reinstein
Head Writer and Editor for Scobol Solo and Masonics (Illinois), TD for New Trier Scobol Solo and New Trier Varsity, Writer for NAQT (2011-2017), IHSSBCA Board Member, IHSSBCA Chair (2004-2014), PACE Member, PACE President (2016-2018), New Trier Coach (1994-2011)

User avatar
First Chairman
Auron
Posts: 3875
Joined: Sat Apr 19, 2003 8:21 pm
Location: Fairfax VA
Contact:

Re: Advice on question writing companies

Post by First Chairman » Fri Dec 19, 2008 9:12 am

Good luck then with your selection process.

I suppose what we can do on another thread or a wiki spreadsheet is to list the major competitions and list the question sources used for those events. This would include a list of all the state championship competitions run by various organizations. But it would be interesting to do that type of comparison, and it would be a great place for people to ask for referrals for the next rounds of bids.

Who's got some free time this holiday break? :wink:
Emil Thomas Chuck, Ph.D.
Founder, PACE
Facebook junkie and unofficial advisor to aspiring health professionals in quiz bowl
---
Pimping Green Tea Ginger Ale (Canada Dry)

User avatar
lasercats
Tidus
Posts: 591
Joined: Fri May 16, 2008 6:11 pm
Location: Tulsa/Norman OK.
Contact:

Re: Advice on question writing companies

Post by lasercats » Wed Dec 31, 2008 3:45 am

I am interested to hear how the selection pans out. The problem with QB in Oklahoma is that sets have to be challenging enough to encourage school like Booker T, Jenks, and Edmond while still being accessible to hillbilly schools who also play (of which there are surprisingly many).
Maggie Larkin
Booker T. Washington '07
University of Oklahoma '11

User avatar
Sir Thopas
Auron
Posts: 1330
Joined: Tue Mar 28, 2006 10:10 pm
Location: Hunter, NYC
Contact:

Re: Advice on question writing companies

Post by Sir Thopas » Wed Dec 31, 2008 9:17 am

lasercats wrote:I am interested to hear how the selection pans out. The problem with QB in Oklahoma is that sets have to be challenging enough to encourage school like Booker T, Jenks, and Edmond while still being accessible to hillbilly schools who also play (of which there are surprisingly many).
I'm...pretty sure that's a problem with QB everywhere.
Guy Tabachnick
Hunter '09
Brown '13

http://memoryofthisimpertinence.blogspot.com/

User avatar
Cheynem
Sin
Posts: 6508
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Advice on question writing companies

Post by Cheynem » Wed Dec 31, 2008 10:55 am

I think it's a problem because, well, for the most part, this is an issue that quiz bowl needs to confront at most regular events, high school and at least novice-level college. Questions should be written in a good, accessible, pyramidal, challenging way to appeal to both the novice and the veteran alike. There certainly have been good examples of this happening.

(I am not arguing for any sort of "dumbing-down" of quiz bowl.)
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
First Chairman
Auron
Posts: 3875
Joined: Sat Apr 19, 2003 8:21 pm
Location: Fairfax VA
Contact:

Re: Advice on question writing companies

Post by First Chairman » Wed Dec 31, 2008 2:37 pm

That is perhaps the hardest challenge to confront because it is extremely difficult to appeal to teams that meet every other day for a couple of hours to those school teams that just met each other on the van/bus trip to the competition. There's no way we can have everyone run the same 5K if some of the competitors have never put on running shoes before. There need to be enough events where the lowliest team feels interested in competing again and getting better without compromising the enjoyment and develop other teams for national-caliber play.
Emil Thomas Chuck, Ph.D.
Founder, PACE
Facebook junkie and unofficial advisor to aspiring health professionals in quiz bowl
---
Pimping Green Tea Ginger Ale (Canada Dry)

Locked