Should detailed statistics be released before a set is clear?

[ryanrosenberg]

As quizbowl begins to regularly collect detailed statistics (e.g. category stats, buzzpoint stats, question-level stats) at tournaments, I wanted to discuss the level of privacy that should be maintained about these more detailed statistics, and if they need be embargoed until the set is clear.

All of them provide some more information into a set than traditional statistics do, but nothing that's incredibly useful. Oftentimes, the sort of information that could be gleaned from detailed statistics is brought up in conversation already (e.g. "Adam was doing a great job on the history," "the science felt tough"). I have a slight inclination towards allowing these detailed stats to be released prior to the end of set mirrors for these reasons, but I can understand the inclination towards security, and am curious to hear other opinions.

[Cheynem]

They definitely should not be released publicly before the set is clear in my opinion. I don't see a problem in letting players who played the tournament privately access them.

[Asterias Wrathbunny]

I think it makes the most sense for writers to get the .qbj or .json files from moderators who used MODAQ (or TDs if they collect them) and view the information locally until a set is clear. I haven't used the stats visualizer app much so I don't know if that could accomplish this but if not, maybe a library could be developed for this purpose. If category-level information is released, I think the best and securest course of action for now is for the writers to post image files of the visualized data. If a web app were to post detailed information before a set is clear, extreme caution should be exercised as resources loaded in the browser can be accessed in full, even if the page is only displaying the category-level information.

[The Ununtiable Twine]

It seems like something that should be kept private until the set is clear. If I was allowed to access the statistics beforehand, I may develop some strange, unnatural disposition as to how to play the questions in certain topics.

[Atlashill]

When devising the category stat components for the latest version of HILLSheets, this was one of the big questions I had to consider. Ultimately, I set it up to where category stats would be compiled and released at the discretion of the set's writer, and only available to the public through a view-only scoreboard. I'd be up for leaving it to the discretion of each set's writer.

As for playing the set differently if stats were accessible from earlier mirrors... how is that different than seeing the average score per round & PPBs as already compiled by more rudimentary systems?

[The Ununtiable Twine]

As for playing the set differently if stats were accessible from earlier mirrors... how is that different than seeing the average score per round & PPBs as already compiled by more rudimentary systems?

Different celerity for some topics as opposed to others suggests that your team might be able to predetermine how aggressively it should play certain topics. Average scores and PPBs per round may suggest that a particular set or round is comparatively difficult, but doesn't necessarily give the team an idea of how to play the set outside of "questions are hard to power on this set/packet" or "this set probably plays pretty slow", whereas if certain topics have somewhat low celerity with respect to others, it can be inferred that those will probably come down to late-question buzzer races more often than not, for instance. Also, writers write questions so that people listen to them. If players know certain topics have lower celerity, they may choose to ignore a lot of clues they would have otherwise listened to, which is disrespectful to the writers in my opinion. Seeing stats ahead of time can also suggest that certain topics' middle clues are easier or that questions in certain topics are easier to power than others. When lots of games and tournaments are ultimately decided by very little, it seems like some games/tournaments could ultimately be decided by being able to infer things from the advanced statistics or attempt to infer things from them (which, let's face it, is unnatural).

It's usually pretty easy for anyone to see how a tournament is going to play overall by either looking at the basic stats or playing a round or two. Therefore, any early advantage a team has based on seeing the usual statistics beforehand is usually cancelled out fairly quickly because most players get a feel for the set after a couple of rounds even if they haven't seen those. Without the advanced information, it's harder to get a feel for how certain topics are going to play after hearing just a couple of questions in them because the data is noisy at best. (Someone in your room might have just had a good buzz on a hard thing or the players may not have known/remembered relatively easy things on certain questions, for instance.) On the other hand, it's significantly easier to do so if you have access to the advanced data. Figuring out how certain topics are playing relative to others mid-tournament is an imperfect process, but has always been a natural part of the game.

Whatever the editors' decision turns out to be, make sure to either not publish the advanced statistics until after the tournament is done or publish them in a public forum for everyone to see (so that, in theory, either no one can form these strategies or everyone has equal chance to do so). If some select group of your friends who intend to play the set get access to these statistics and their opponents don't, I'd consider that to be fairly dishonest.