2018 Sun God Invitational Detailed Stats

[Jason Cheng]

Here are links to the detailed stats of every mirror that chose to use them. They might still have some issues with bonus answer lines or categorization that we will be fixing this week. I will add links here to each additional mirror as they use them. Please do not share with anyone still yet to play the tournament, or post these in a public forum.

Playtest Mirror, 09/29/2018 - the set changed significantly between the playtest mirror and the live release
Discord

11/10/2018
Combined Stats - across all sites
Johns Hopkins University
New College Florida
Rutgers University
University of Illinois Urbana-Champaign

11/17/2018
Combined Stats - across all sites. The set was changed from 11-10, mostly to make a large number of bonuses easier and correct some errata, but also to replace a couple of tossups and fix clues.
Mizzou Scrimmage
University of Toronto
UCSD

11/24/2018
University of Southampton

12/01/2018
Combined - across all sites.
Brown
Stanford
UT-Austin

[aseem.keyal]

Hey everyone,

Major apologies to the moderators of yesterday's mirrors who were confused why their spreadsheets were not editable for a 5 minute period during the tournament - this is my fault because I changed something without telling Ophir and without understanding how Google Drive handles permissions. Sorry for those who were affected and frustrated by it.

[The Story of a Head That Fell Off]

What do the .5 in the Heard column of the Categories tab mean?

EDIT: Ophir told me, thanks

[aseem.keyal]

Attached is a zip of the packets.

[Jason Cheng]

The first post has been updated with 11/17/2018 site detailed stats. The pdfs .zip file for the same weekend is attached here. Please do not share this zip with anyone--whoever wants them should be directed to this post, should they have access to this discussion forum.

[CadenPetrosian]

The buzz on "sandbars" was incorrectly attributed to me. Josh Lane got that tossup.

[Smuttynose Island]

I wrote some janky code to compute BPAs for all the mirrors with advanced stats. I'm working on generating category BPAs and visual aids. You can find my work here. At the moment I don't promise 100% accuracy in any of these computations.

EDIT: I made a small change to how I compute BPA, ignoring buzzes at the end of the question as they don't contribute area under the cumulative distribution.

EDIT: Fixed an issue where it was counting negs towards the cumulative get count.

[Smuttynose Island]

I went in a different direction and wrote code to simulate games based upon two team's buzzpoint distribution, Neg Rate, and PPBs. I've gone ahead and computed head to head winning percentages for the top 5 teams at the UIUC mirror site. This was accomplished by "playing" each team against each other 1000 times.

Code: Select all

	MSU	NU A	Chi A	OSU	Ill A
MSU	Null	0.706	0.237	0.471	0.737
NU A	0.294	Null	0.169	0.372	0.633
Chi A	0.763	0.831	Null	0.666	0.853
OSU	0.529	0.628	0.334	Null	0.674
Ill A	0.263	0.367	0.147	0.326	Null

Although buzz points are site-specific, I also decided play each of the site winners against each other and compute head to head winning percentages:

Code: Select all

	Team 3	UCLA     Tor A	MSU	Colum	UF A	UMD A
Team 3	Null	0.922	0.84	0.658	0.783	0.811	0.568
UCLA	0.078	Null	0.291	0.135	0.221	0.271	0.075
Tor A	0.16	0.709	Null	0.179	0.303	0.401	0.133
MSU	0.342	0.865	0.821	Null	0.576	0.619	0.346
Colmbia	0.217	0.779	0.697	0.424	Null	0.638	0.176
UF A	0.189	0.729	0.599	0.381	0.362	Null	0.149
UMD A	0.432	0.925	0.867	0.654	0.824	0.851	Null

Eventually I'd like to use these results to create some sort of power ranking for teams that played Sun God. In the meantime, feel free to shoot me any questions about my methods or results of simulations that you want to see.

EDIT: Here are the results of a massive round robin between many of the teams in the Preseason Top 25. Again, each head to head winning percentage was computed by simulating 1000 games between the two teams. Given the site specific nature of buzzpoints and occasional lack of full rosters, I wouldn't put too much stock in these numbers.

Code: Select all

 	OSU	UF A	JHU A	UMD A	Colmbia	MSU	Chi A	UVA	Chi B	NU A	Tornto	UMD B	Mean
Chi A	0.689	0.904	0.907	0.558	0.862	0.777		0.782	0.81	0.852	0.916	0.775	0.802909091
UMD A	0.729	0.831	0.854		0.786	0.688	0.442	0.712	0.722	0.754	0.87	0.676	0.733090909
OSU		0.724	0.727	0.271	0.686	0.547	0.311	0.61	0.585	0.642	0.755	0.498	0.577818182
UVA	0.39	0.731	0.714	0.288	0.667	0.522	0.218		0.579	0.643	0.739	0.509	0.545454545
MSU	0.453	0.602	0.743	0.312	0.566		0.223	0.478	0.473	0.696	0.827	0.56	0.539363636
Chi B	0.415	0.713	0.7	0.278	0.687	0.527	0.19	0.421		0.628	0.748	0.5	0.527909091
UMD B	0.502	0.655	0.648	0.324	0.604	0.44	0.225	0.491	0.5	0.59	0.694		0.515727273
Colum	0.314	0.637	0.634	0.214		0.434	0.138	0.333	0.313	0.548	0.685	0.396	0.422363636
NU A	0.358	0.518	0.506	0.246	0.452	0.304	0.148	0.357	0.372		0.726	0.41	0.399727273
UF A	0.276		0.597	0.169	0.363	0.398	0.096	0.269	0.287	0.482	0.637	0.345	0.356272727
JHU A	0.273	0.403		0.146	0.366	0.257	0.093	0.286	0.3	0.494	0.633	0.352	0.327545455
Tornto	0.245	0.363	0.367	0.13	0.315	0.173	0.084	0.261	0.252	0.274		0.306	0.251818182

[Lake Winnipesaukee Mystery Stone]

Could a link to the Southampton stats be added to the first post?

[Jason Cheng]

Could a link to the Southampton stats be added to the first post?

Done!

[a bird]

I went in a different direction and wrote code to simulate games based upon two team's buzzpoint distribution, Neg Rate, and PPBs. I've gone ahead and computed head to head winning percentages for the top 5 teams at the UIUC mirror site. This was accomplished by "playing" each team against each other 1000 times.

..results...

Eventually I'd like to use these results to create some sort of power ranking for teams that played Sun God. In the meantime, feel free to shoot me any questions about my methods or results of simulations that you want to see.

EDIT: Here are the results of a massive round robin between many of the teams in the Preseason Top 25. Again, each head to head winning percentage was computed by simulating 1000 games between the two teams. Given the site specific nature of buzzpoints and occasional lack of full rosters, I wouldn't put too much stock in these numbers.

This is a cool project that shows the usefulness of advanced stats. While I'm interested in seeing results including recent mirrors, I want to add another warning. The editors made significant changes after the first round of mirrors, and difficulty tweaks could have a significant impact on stats from different iterations of the set.

[Smuttynose Island]

I went in a different direction and wrote code to simulate games based upon two team's buzzpoint distribution, Neg Rate, and PPBs. I've gone ahead and computed head to head winning percentages for the top 5 teams at the UIUC mirror site. This was accomplished by "playing" each team against each other 1000 times.

..results...

Eventually I'd like to use these results to create some sort of power ranking for teams that played Sun God. In the meantime, feel free to shoot me any questions about my methods or results of simulations that you want to see.

EDIT: Here are the results of a massive round robin between many of the teams in the Preseason Top 25. Again, each head to head winning percentage was computed by simulating 1000 games between the two teams. Given the site specific nature of buzzpoints and occasional lack of full rosters, I wouldn't put too much stock in these numbers.

This is a cool project that shows the usefulness of advanced stats. While I'm interested in seeing results including recent mirrors, I want to add another warning. The editors made significant changes after the first round of mirrors, and difficulty tweaks could have a significant impact on stats from different iterations of the set.

This is an great point that's always worth keeping in mind. It is one reason why I avoided comparing any of the Discord teams to collegiate teams.

Another consideration that I overlooked was the impact of bad data on both this project and computing BPA. I'll have more to say in the "Introducing BPA" thread, but it is very easy for unrecorded TUs (ones where no buzzpoint is recorded) to drastically alter BPA as well as the computation of a team's buzz distribution. There are some obvious workarounds, but I don't think any of the current computations of BPA employ them.

After UO Finals, I hope to update the Sun God BPA as well as this round robin with the UK and California mirror stats. Longer term, I'd like to see how robust advance stat metrics are at predicting future tournament success. Is a large-scale simulated round robin better at predicting ACF Nationals placement than the A-value? How about ACF Regionals performance? Are advance stats really that site dependent? How can survival statistics help compute a team's buzzpoint distribution? These are all interesting questions, but ones that unfortunately require a bit more data and time to answer.

[Jason Cheng]

OP has been updated with 12/01 sites.

At some point, we may publish a changelog, but in general, science bonuses (bio/chem) were edited down in difficulty, and some number (not as many, but not insignificantly small) of other questions were changed or replaced based on feedback/errata over the last three weeks or so. Not much else was changed structurally besides specifically problematic questions (helices was replaced with the finals Western blots tossup, which itself was replaced with a tossup on proline, illegal immigrants/undocumented residents tossup was changed for a more targetable answerline i.e. illegal immigration, etc).

The set changed significantly between the playtest mirror and the first live mirror (20-25% of it was written during that time period).

[Jason Cheng]

Packets from 12/01 mirrors:

sgi-pdfs-2018-12-01.zip

(3.58 MiB) Downloaded 248 times

[Protean]

I went in a different direction and wrote code to simulate games based upon two team's buzzpoint distribution, Neg Rate, and PPBs.

After UO Finals, I hope to update the Sun God BPA as well as this round robin with the UK and California mirror stats. Longer term, I'd like to see how robust advance stat metrics are at predicting future tournament success. Is a large-scale simulated round robin better at predicting ACF Nationals placement than the A-value? How about ACF Regionals performance? Are advance stats really that site dependent? How can survival statistics help compute a team's buzzpoint distribution? These are all interesting questions, but ones that unfortunately require a bit more data and time to answer.

This is super cool and I really look forward to this work. I was just wondering if you still had plans to do the category BPAs and visual aids? Thanks!

[hftf]

Static visualizations are now available.

Because SGI is not clear, access is restricted (username: oscar, password: espla). You must obtain access via this private forum.

[Cody]

Static visualizations are now available.

Because SGI is not clear, access is restricted (username: oscar, password: espla). You must obtain access via this private forum.

amazing, Ophir. amazing.

[A Dim-Witted Saboteur]

Static visualizations are now available.

Because SGI is not clear, access is restricted (username: oscar, password: espla). You must obtain access via this private forum.

These visualizations are excellent. Thanks for making them!