The PACE Question Set Certification Program and Me

[Stained Diviner]

Here is a timeline of my experiences with the PACE Question Set Certification Program.

10/30: I emailed PACE asking them if they would consider certifying Solo and Masonics.
11/11: I emailed them again, since they never got back to me.
11/18: I emailed them again, this time cc'ing several people since they never got back to me.
11/19: I sent them the sets, since they told me they would consider them.
11/25: I got an email asking me why the 3rd Tiebreakers in Solo were so short. I replied the same day.
12/3: I got an email saying that I would hear something soon.
12/4: I got the message below from Mike.
12/28: I sent a reminder that PACE was supposed to get back to me about Masonics a few weeks earlier.
1/3: I sent this timeline to PACE and reminded them that I still had not heard back about Masonics. I got an email back saying that they will get to Masonics by Sunday and that they have a lot to do to improve the program. When I asked them if it would be better for them to publicly acknowledge the shortcomings of the program or for me to do it, I was told that I am welcome to post my opinion.

Just to be clear, I don't have a problem with them turning down Solo, and I won't have a problem when they turn down Masonics. I just want to make sure that nobody is under the false impression that this program is being managed well. Since PACE doesn't want to say it, you're stuck with this post.

>>>
Here is the message I got on 12/4:

Hi David, we are unable to certify the Scobol Solo set as exemplary. There are many great questions in this tournament and it's clearly an above average set. However, we had the following concerns that prevented us from ranking the set as exemplary:

1. Question length - The questions seemed quite long compared to a regular high school quizbowl tournament.
2. Tossup difficulty - This may be a result of having to produce such a large number of tossups compared to a regular tournament. Some examples include r-k, closed, EVT, Miro, and San Martin. One suggestion we have in this area is to not shy away from repeatedly asking about figures in the core canon. Rather than asking about Miro, for instance, you could ask about Van Gogh's Starry Night and Van Gogh's Night Cafe as separate tossups. Some other questions had accessible answer lines but were very top heavy with the clues, such as the Beethoven tossup.

There were a few other minor issues around things like tightening up some of the language (removing clauses like "it is thought to be"), but the two reasons above were the main issues that prevent us from ranking this set as exemplary.

We will try to get back to you within a week or so about Masonic. We're a little swamped with some other sets at the moment.

(As a parenthetical note, I'll point out that the phrase "it is thought to be" does not appear in the set. This is not important--I am only saying it because I find it amusing.)

[Dominator]

I find this all very disappointing. After the debacle that was NSC 2011, PACE managed to restore confidence with an excellently-run and -written NSC 2012. Announcing a new program this year and immediately mismanaging it comprises that newfound confidence.

I love the idea for the program, but so far it has (1) been unable or unwilling to define "exemplary" beyond giving a checklist of bad things that all tournaments should avoid, and (2) been unable to deliver on the deadlines they set and which they would have to meet if they have any hope that their designations would inform decisions about what question sets tournaments should use.

[Stained Diviner]

I'll add that the GSAC set has issues. Any TD thinking of using it should make sure they have permission to edit it and should contact me and Jonah with plenty of time before their tournament. We sent them general comments to improve grammar, include at least some pronunciation guides, not use bonuses that vary ridiculously in difficulty, list alternative answers that are correct or promptable, and relate parts within bonuses, and we gave them specific feedback on 40 questions that had problems not covered by those general comments.

I don't know which version of it was certified. I can only hope that it wasn't the version that was sent to New Trier.

[TheDoctor]

This confused me, as well. I don't see the GSAC set as having significantly fewer faults than this year's Solo, and I'm still baffled by exactly what distinguishes an "above average" set from an "exemplary" one for those operating the program. What I am seeing happen is really only a confirmation of my fear that subjectivity in evaluation would not be guarded against.

[Mike Bentley]

We would like to apologize for the delay in turning around this set. In particular, the Masonics set slipped under the radar due to my oversight. This is our first year running this program and we're still working out the best process for getting the certifications done in a timely manner.

Some rumors have gone around that this certification program hasn't looked at any sets, or has not been providing useful information to those who submit. This is absolutely not the case. We have completed reviews of five full question sets this year, are underway with a sixth, and have rated two of those sets exemplary thus far. Each of those sets has gotten (or will get) an email response from us with some pointers about general trends in the set that could be improved for future uses, or skills to work on in future writing projects.

In the bigger picture, this is a macro-level program which seeks to improve the average quality of writing standards across high school quizbowl nationwide. We are confident that this process of diffusing writing advice and higher, more coherent standards through the quizbowl community is still happening by way of the program this year.

In terms of the standards laid out in the first post, the notion that the standards are "vague" or "ill-defined" is similarly unfounded. The program is looking at four aspects of question writing which are easily discernible and easily defined, as indeed they were in the initial announcement, and a set is exemplary if it consistently succeeds at that aspect of question writing across the entire body of questions. Those aspects are, and have been, the following: length, difficulty, pyramidality/clue quality, and comprehensibility/legibility. We apply those standards roughly as follows: If a general trend is apparent across a wide swath of the set that one of these factors is inadequate, it becomes much more difficult to label a set exemplary; if multiple general trends show two or more of these factors to be inadequate, the set is not exemplary.

Some examples can serve to illustrate how these are applied if this is still difficult to understand. If, for example, a set written in 10-point Times New Roman or thereabouts has many tossups that go well over the sixth line, or many bonus parts over the second line, or uses a lot of excess verbiage all over the place (e.g. "He is most remembered today for discovering" vs. "He also discovered") it's pretty clear that the set has an issue with questions that are too long. If, for example, all of the science is wildly difficult for a high school audience but other categories are largely fine, that discrepancy counts against the Difficulty criterion, as does a set where a large number of questions in all categories are too hard. A preponderance of incorrect clues across many questions will threaten Exemplary status for the pyramidality/clue quality criterion, and we already noted in the first post that a set will also miss this criterion if clues are out of order, or it is evident to experienced editors that too many clues have older question sets and Wikipedia as their source without other verification of their importance. Comprehensibility and legibility are just that - a set does an exemplary job if the questions in the set make coherent sense when they are read aloud (as many are during the certifying process).

Because the committee has been spotting general trends within these criteria, so individual outliers will not alter the rating of a set. (If there is one eight-line tossup, for example, or if three or four questions have bizarre syntax, but the others are fine, the set can still be Exemplary.)

Those who would like more exact examples of such general trends, or with a specific curiosity about one kind of general trend, are free to ask publicly or in private.

I will also state that we stand by our rating of the GSAC set. Furthermore, as I mentioned in my e-mail to Mr. Reinstein, we also felt that Solo was a good set with many excellent questions, but it fell just short of the exemplary status due to the question length and tossup difficulty.

An additional note: It is in part because PACE has been focusing on making the 2013 NSC a success that this program is running more slowly than planned. Members of our organization - committee members included - are ensuring first and foremost that the 2013 NSC is written, edited, and run as well as last year's. At many points during this past semester, that NSC-specific work has had to come first. As such, we do not expect the existence or non-existence of this program to adversely affect the efficient direction of our championship in any way whatsoever.

PACE will continue to accept submissions to the Question Set Certification Program for the 2012-13 school year, and will continue to do its best to rate them in a timely manner. The program is optional, and those who do not wish to send in their sets, for whatever reason, do not have to do so.

We will be reviewing the procedure and effectiveness of the program amongst ourselves at the end of this season, and will only announce that we are running this program in 2013-14 if are confident that it is once again something we can do well to the benefit of the high school circuit as a whole.

Let me conclude by once again apologizing to Mr. Reinstein for the unacceptable delays in getting feedback back to him. I will personally make sure that we finish evaluating the Masonics set by tomorrow, and that for the remainder of the year we have a faster turnaround time on new sets coming in.

[alinktothefuture]

I will also state that we stand by our rating of the GSAC set.

The GSAC XX editors believe that PACE's rating speaks for itself, but if prospective mirror hosts or TDs of scheduled mirrors have any questions or would like to see a sample of the set, please feel free to contact us at gsacdirector AT gmail DOT com.

EDIT: Quote added.

[TheDoctor]

I will also state that we stand by our rating of the GSAC set. Furthermore, as I mentioned in my e-mail to Mr. Reinstein, we also felt that Solo was a good set with many excellent questions, but it fell just short of the exemplary status due to the question length and tossup difficulty.

The problem, as I see it, is that Solo seems to have been evaluated without regard to how the set is conceived and marketed annually. This is meant to be a set that is challenging to high-level players and discerns adequately between them which is the best. The expectation of this set's market and the tradition in which the set is produced were clearly not taken into account. Rather, it seems to have been judged in the context of a regular-difficulty (this by itself is badly defined at best), team-oriented tournament, which sets it immediately in poor light.

It is for this reason that I refused to allow PACE to evaluate this year's SCOP set; I don't believe a program like this is capable of fairly evaluating any set not produced to be regular difficulty. Mike claims

PACE will continue to accept submissions to the Question Set Certification Program for the 2012-13 school year, and will continue to do its best to rate them in a timely manner. The program is optional, and those who do not wish to send in their sets, for whatever reason, do not have to do so.

and this is true but, in the spirit of the original post, I would like to share my own experience with the PACE Question Set Certification Program.

14 Oct: I was contacted by PACE, who requested SCOP3 for evaluation. I don't know how widespread it was for PACE to contact set editors, but I was later surprised to learn that it had not happened to several other editors with whom I am acquainted. I was (and am) stymied as to why they were so keen to evaluate a novice set, since their primary focus and greatest utility is clearly to regular difficulty, and set aside some time to consider.
16 Oct: I was again contacted, this time by a different member of PACE, with a very insistent email pressuring me to participate.
17 Oct: I responded in the negative, and explained that I did not feel a novice set would be appropriate for evaluation and that, as most of our mirrors had already happened, the evaluation would not do us much good.
17 Oct: PACE responded, countering my explanation and again offering to evaluate the set.
17 Oct: I again refused.

This exchange may shed some light on why I have low confidence in the program as a whole. In light of what happened to Solo, I remain happy that I did not submit my (non-standard, non-regular-difficulty) set to the program. I do not believe the program as a whole to be capable of taking into account the vastly different philosophy that must necessarily go into producing a novice set, and from the program's interaction with Solo, I have begun to feel that this applies to any even slightly non-standard tournament.

[Irreligion in Bangladesh]

It is in part because PACE has been focusing on making the 2013 NSC a success that this program is running more slowly than planned. Members of our organization - committee members included - are ensuring first and foremost that the 2013 NSC is written, edited, and run as well as last year's. At many points during this past semester, that NSC-specific work has had to come first. As such, we do not expect the existence or non-existence of this program to adversely affect the efficient direction of our championship in any way whatsoever.

I absolutely believe that NSC won't be adversely affected by this program. That said, it just seems like the above is a great reason to avoid taking on other responsibilities (particularly time-sensitive ones like this) until PACE finds staff whose time can be devoted solely to non-NSC pursuits.

[Stained Diviner]

GSAC questions were about 590 characters, and Solo questions were about 660. If the upper limit for acceptable length is somewhere between 590 and 660, and one of the things you are trying to deny is that your standards are vague, then you should say what an acceptable length is.

[Mike Bentley]

I will also state that we stand by our rating of the GSAC set. Furthermore, as I mentioned in my e-mail to Mr. Reinstein, we also felt that Solo was a good set with many excellent questions, but it fell just short of the exemplary status due to the question length and tossup difficulty.

The problem, as I see it, is that Solo seems to have been evaluated without regard to how the set is conceived and marketed annually. This is meant to be a set that is challenging to high-level players and discerns adequately between them which is the best. The expectation of this set's market and the tradition in which the set is produced were clearly not taken into account. Rather, it seems to have been judged in the context of a regular-difficulty (this by itself is badly defined at best), team-oriented tournament, which sets it immediately in poor light.

It is for this reason that I refused to allow PACE to evaluate this year's SCOP set; I don't believe a program like this is capable of fairly evaluating any set not produced to be regular difficulty. Mike claims

PACE will continue to accept submissions to the Question Set Certification Program for the 2012-13 school year, and will continue to do its best to rate them in a timely manner. The program is optional, and those who do not wish to send in their sets, for whatever reason, do not have to do so.

and this is true but, in the spirit of the original post, I would like to share my own experience with the PACE Question Set Certification Program.

14 Oct: I was contacted by PACE, who requested SCOP3 for evaluation. I don't know how widespread it was for PACE to contact set editors, but I was later surprised to learn that it had not happened to several other editors with whom I am acquainted. I was (and am) stymied as to why they were so keen to evaluate a novice set, since their primary focus and greatest utility is clearly to regular difficulty, and set aside some time to consider.
16 Oct: I was again contacted, this time by a different member of PACE, with a very insistent email pressuring me to participate.
17 Oct: I responded in the negative, and explained that I did not feel a novice set would be appropriate for evaluation and that, as most of our mirrors had already happened, the evaluation would not do us much good.
17 Oct: PACE responded, countering my explanation and again offering to evaluate the set.
17 Oct: I again refused.

This exchange may shed some light on why I have low confidence in the program as a whole. In light of what happened to Solo, I remain happy that I did not submit my (non-standard, non-regular-difficulty) set to the program. I do not believe the program as a whole to be capable of taking into account the vastly different philosophy that must necessarily go into producing a novice set, and from the program's interaction with Solo, I have begun to feel that this applies to any even slightly non-standard tournament.

One of the main problems we're trying to solve with the question certification program is the proliferation of house-written sets that are overly difficult for the vast majority of teams. We want to use this program to recognize house-written sets that keep the difficulty under control and provide a great experience for all teams.

In the original announcement, we mentioned that sets written for upper-echelon teams would not be considered exemplary. Although it's certainly possible for these sets to be well written and all parties aware of what they're getting into ahead of time, we've found that in many cases, sets of this difficulty end up being played in fields where teams would be much better served with easier sets.

Since we have not seen the opposite problem of too many house-written sets undershooting their difficulty level, we also said in the announcement post that we would be willing to grant exemplary status to novice sets, provided they meet the other criteria.

We have made an effort to contact all people writing a house-written set. We were not more keen on evaluating the SCOP Novice set than any other set out there. If an editor hasn't been contacted, that is an oversight on our part, and not due to some bias of wanting to evaluate some sets and not others.

[Dominator]

We have made an effort to contact all people writing a house-written set. We were not more keen on evaluating the SCOP Novice set than any other set out there. If an editor hasn't been contacted, that is an oversight on our part, and not due to some bias of wanting to evaluate some sets and not others.

I was the first person to announce plans for a housewrite for this season and I was never contacted.

[Stained Diviner]

I guess it's hard to stay mad when they apologize and finally do get back to me. I can only hope that PACE's timing with me is the exception rather than the rule.

If anybody else is applying, they should keep in mind that PACE seems to be making both low difficulty and short length very high priorities when judging sets. On the one hand, I can respect the fact that those things are important, but on the other hand I am weary that they might be placing those priorities ahead of or on the same footing of questions having useful clues and bonus difficulty being even, things that help knowledgeable teams win matches and thus make results valid.

I'll add that Donald Taylor wrote close to half of the Masonic questions, and Jonah and Egan gave a lot of feedback on them. Looking at the feedback from PACE, the praise was focused on Donald's questions. I'll also add that PACE only reviewed the Sectional questions. I am not going to have them review the State questions, because those would be judged too difficult. I am not going to post PACE's comments on the questions, because I would have to redact half of the comments to preserve question integrity.

The reason I asked PACE if they were interested in reviewing my questions before just sending them in is that neither set is mACF format. I would have been fine with PACE not even considering the sets due to the odd circumstances of both of them. Given a choice between the sets, I would favor Solo over Masonics because the Solo distribution is better. So it goes.