Crowdsourced Head to Head Rankings

Old college threads.
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Crowdsourced Head to Head Rankings

Post by sephirothrr »

While trying to rank my top 25 for Mike's midseason poll, I came upon a small problem - I'm really bad at ranking things in a list. To that end, I cobbled this little thing together.

Presenting the creatively named PriyaRank!

While I had designed it to help me find my top 25, I realized that one person alone would take a long time to simulate the necessary match results.

Since Quizbowl lives by the idea that the only way to rank teams is to throw them all in a giant round robin, that's just what I did!
The current rankings are on the home page, and you can contribute in the arena by clicking "Rank" and choosing a winner for a hypothetical match.

Over time, assuming good faith choices, the system will gradually approach what people believe to be the "true" rank, so I'm curious to see how this little experiment goes.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
User avatar
vinteuil
Auron
Posts: 1454
Joined: Sun Oct 23, 2011 12:31 pm

Re: Crowdsourced Head to Head Rankings

Post by vinteuil »

If nothing else, I suspect that playing around with this will help prevent people forgetting about teams/having regional availability bias when picking lower-ranked teams on their ballots.
Jacob R., ex-Chicago
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

Interestingly enough, there's already a strong showing of people not being aware of the strengths of teams in various regions, especially in the middle of the pack.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
Ringil
Rikku
Posts: 412
Joined: Sun Sep 21, 2008 12:46 am

Re: Crowdsourced Head to Head Rankings

Post by Ringil »

This is lots of fun.
Libo
Washington '14, Michigan '13, Troy High School '09
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Crowdsourced Head to Head Rankings

Post by theMoMA »

I think you're probably neutering the true wisdom of the crowd a little bit by having the current record display below each team. Relative independence of opinions is important to getting better data in this kind of ranking.
Andrew Hart
Minnesota alum
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

theMoMA wrote:I think you're probably neutering the true wisdom of the crowd a little bit by having the current record display below each team. Relative independence of opinions is important to getting better data in this kind of ranking.
Yeah, I definitely noticed that, so that's gone now.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
User avatar
Excelsior (smack)
Rikku
Posts: 386
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Crowdsourced Head to Head Rankings

Post by Excelsior (smack) »

This is awesome! (In retrospect, I can't believe nobody had thought of this before, seeing as it makes _so_ much sense!)
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Crowdsourced Head to Head Rankings

Post by theMoMA »

One possible drawback (and, of course, drawback to every ranking system) is imperfect information about lineups. If all rankers were considering "Minnesota" to be a team with Mike Cheyne on it, as it may be for ICT, I think that Minnesota would do a lot better in the ranks. But I agree that this is a very, very good idea that, in my opinion, has already produced very credible results.
Andrew Hart
Minnesota alum
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: Crowdsourced Head to Head Rankings

Post by Rococo A Go Go »

theMoMA wrote: If all rankers were considering "Minnesota" to be a team with Mike Cheyne on it, as it may be for ICT, I think that Minnesota would do a lot better in the ranks.
Considering I've voted against Minnesota most times they've come up, this is probably a pertinent point.
Nicholas C
KQBA member
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: Crowdsourced Head to Head Rankings

Post by AKKOLADE »

This is cool.

Are you goofy enough to do this for high schoolers?
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

This deployment was really just a Phase I proof of concept. By the end of the day I should be ready to deploy Phase II, which is more in line with what I believe are the goals of the original rating system - namely, records and ratings will be hidden until Mike Cheyne publishes his results, at which point I shall publish mine. Hopefully, that will allow people to choose winners based on their own opinion of relative skill and not possibly incorrect trends in the data. (read: trolls)
Grams's Go-Go Boots wrote:This is cool.

Are you goofy enough to do this for high schoolers?
It wouldn't be any extra work on the software side, if, say, someone wanted to provide me with a text file of relevant teams, one team per line... :wink:
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
jonah
Auron
Posts: 2383
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: Crowdsourced Head to Head Rankings

Post by jonah »

I just sent you a list of all the schools that have entered an NAQT tournament this year for which we have results.
Jonah Greenthal
National Academic Quiz Tournaments
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: Crowdsourced Head to Head Rankings

Post by Rococo A Go Go »

A pool of hundreds of high school teams is going to lead to a lot of people making selections on scant information. There may be some interesting results however.
Nicholas C
KQBA member
Tanay
Rikku
Posts: 427
Joined: Wed Jun 03, 2009 5:05 pm

Re: Crowdsourced Head to Head Rankings

Post by Tanay »

Sulawesi Myzomela wrote:A pool of hundreds of high school teams is going to lead to a lot of people making selections on scant information. There may be some interesting results however.
In that case, it might be better just to include teams that have signed up for HSNCT or NSC.
Tanay
ex-Berkeley, ex-Bellarmine
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: Crowdsourced Head to Head Rankings

Post by AKKOLADE »

I could just post my stat file.
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

jonah wrote:I just sent you a list of all the schools that have entered an NAQT tournament this year for which we have results.
Sweet! However, while setting that up, I made a realization - with that many teams, my purely random approach to generating matches would fail due to the staggeringly high number of simulation matches needed. Unfortunately, the HSNCT field will have a similar effect. While that's a problem I intend to fix, in the meantime, I'm looking to cap the field size at 100 teams. However, I know very little about the high school quiz bowl landscape, so I've decided to take the top 100 teams from Fred's HSQBRank, unless anyone more knowledgeable than me can offer a suggestion.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: Crowdsourced Head to Head Rankings

Post by AKKOLADE »

You could also consider doing something like this: http://www.basketball-reference.com/about/elo.html in terms of match-up control.
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: Crowdsourced Head to Head Rankings

Post by The Ununtiable Twine »

There are so many interesting things that could come of this, say if you make everyone make an account and give some additional info to sign into this. With enough data on each individual voter you can surely weed out trolls. You can also develop a national ranking as well as regional ones, and...get this...we may finally have a way to measure regional bias and prove that it exists and to what degree! I still think you should start with a blank slate and not show the results to anyone, then compare to the Cheyne poll. You can share the results of certain individuals that participated in both PriyaRank and the Cheyne poll to see if their ballots were indeed consistent with how they participated in PriyaRank to get even more interesting results. This is a rather fantabulous thing and I love it very very much.

EDIT: You could even ask people to tell you how much emphasis they put on ACF/mACF with respect to NAQT or you could even ask who would win on NAQT or ACF/mACF. There are so many things that you can do to expand this and get super-interesting data.
Jake Sundberg
Louisiana, Alabama
retired
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: Crowdsourced Head to Head Rankings

Post by The Ununtiable Twine »

Additionally, it might be noted that Yogesh plays for New Mexico State so I doubt they're the second-worst team here. A complete list of rosters somewhere may also help with this - perhaps you could make that available on your site?
Jake Sundberg
Louisiana, Alabama
retired
User avatar
Ndg
Rikku
Posts: 387
Joined: Tue Oct 30, 2012 3:24 pm
Location: Wisconsin

Re: Crowdsourced Head to Head Rankings

Post by Ndg »

The Ununtiable Twine wrote:Additionally, it might be noted that Yogesh plays for New Mexico State so I doubt they're the second-worst team here. A complete list of rosters somewhere may also help with this - perhaps you could make that available on your site?
Rosters would definitely be a good thing to show. Even better would be to have some sort of statistical summary like on the Basketball-Reference and Baseball-Reference Elo Raters, although I don't know how much effort it would take to maintain something like that.

As is, this is still a really neat thing to have.
Andrew Nadig

Mannhiem Mannheim Manheim Township, 2005-11
Carnegie Mellon University, 2011-15
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: Crowdsourced Head to Head Rankings

Post by AKKOLADE »

Ndg wrote:
The Ununtiable Twine wrote:Additionally, it might be noted that Yogesh plays for New Mexico State so I doubt they're the second-worst team here. A complete list of rosters somewhere may also help with this - perhaps you could make that available on your site?
Rosters would definitely be a good thing to show. Even better would be to have some sort of statistical summary like on the Basketball-Reference and Baseball-Reference Elo Raters, although I don't know how much effort it would take to maintain something like that.

As is, this is still a really neat thing to have.
I agree with both points on this post.
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: Crowdsourced Head to Head Rankings

Post by The Ununtiable Twine »

Well, it looks like you'll have to find a different twitter handle for PriyaRank.

https://twitter.com/priyarank

What the hell does any of this mean? I wish Jerry were around to translate - I see something about Angry Birds...Angry Russian Birds.
Jake Sundberg
Louisiana, Alabama
retired
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: Crowdsourced Head to Head Rankings

Post by The Ununtiable Twine »

Dartmouth (39), Louisville (47), UVA (10), Penn (9) be gettin' trolled! Meanwhile, someone also hates Rob and Bernadette (32)! Could someone at Vassar (24) or Claremont (15) possibly be behind this???
Last edited by The Ununtiable Twine on Tue Feb 25, 2014 1:04 am, edited 1 time in total.
Jake Sundberg
Louisiana, Alabama
retired
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: Crowdsourced Head to Head Rankings

Post by Rococo A Go Go »

The Ununtiable Twine wrote:someone also hates Rob and Bernadette (32)
This sounds like it could be a...

Conspiracy theory.
Nicholas C
KQBA member
User avatar
Good Goblin Housekeeping
Auron
Posts: 1100
Joined: Sun May 23, 2010 10:03 am

Re: Crowdsourced Head to Head Rankings

Post by Good Goblin Housekeeping »

This is really neat! I'm a bit confused on a few things though, namely how stuff like this happens

Image
Andrew Wang
Illinois 2016
bradleykirksey
Wakka
Posts: 187
Joined: Sat Nov 12, 2011 5:09 pm

Re: Crowdsourced Head to Head Rankings

Post by bradleykirksey »

The Ununtiable Twine wrote:There are so many interesting things that could come of this, say if you make everyone make an account and give some additional info to sign into this. With enough data on each individual voter you can surely weed out trolls.

EDIT: You could even ask people to tell you how much emphasis they put on ACF/mACF with respect to NAQT or you could even ask who would win on NAQT or ACF/mACF. There are so many things that you can do to expand this and get super-interesting data.

I agree with both of these things. I was tempted to help UCF out by saying we'd beat Yale, but I didn't. I might just be an exceptionally bad person, but it was tempting. And having done surveys for undergrad research, I can promise you that there are people sad and lonely enough to lie for the fun of it. Maryland B being ranked 3 above Maryland A might be an example of not enough people voting, but it might also be that.

I also think that some teams are a lot better at ACF than NAQT. We played Alabama very close a few years back on NAQT tossups (losing on the last tossup of course). Then when they removed a lot of American current events and Jesus tossups and geography, we got crushed at ACF. It might be fun to try to measure A) how much better teams actually are at one format than the other and B) how people perceive how much better some teams are than others on one format than the other.

I really do like the system, and I'll be interested to see how this turns out at the end when it evens out.

And then there's also the fact that you have people like me rating, and really, why would I know if Minneanapolis CTC is better than Alabama? I mean, I feel like they are (and honestly that maybe we shouldn't be ahead of either but maybe should be ahead of Vasser or WUSTL, but honestly I've never seen Rob Carson or any of the Vasser players in my life and I wouldn't know.) There's only a handful of players who have been at enough national tournaments to really say this for sure.

But like I said, I do like the system and I'll be very interested in where it goes.

Edit: Since I started that reply, Yale took a nose dive and Virginia isn't moving up. UCF is in the top 20. I think that's trolls at work?
Bradley Kirksey
Mayor of quiz bowl at the University of Central Florida (2010-2015)
The club at Reformed Theology Seminary Orlando (2017 - 2021)
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: Crowdsourced Head to Head Rankings

Post by The Ununtiable Twine »

bradleykirksey wrote: I also think that some teams are a lot better at ACF than NAQT. We played Alabama very close a few years back on NAQT tossups (losing on the last tossup of course).
That was at a combined field on D2 questions, but it was a very fun match indeed!

EDIT: Perhaps I could attribute this to your team having an unusual power spike and much higher PPB than in previous rounds, but I am not Joseph Nation and hence I will not overanalyze the results of a combined-field SCT or attribute that close game to anything other than that particular game being close, however many Jesus questions may or may not have been in the packet.
Last edited by The Ununtiable Twine on Tue Feb 25, 2014 12:18 am, edited 2 times in total.
Jake Sundberg
Louisiana, Alabama
retired
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: Crowdsourced Head to Head Rankings

Post by Rococo A Go Go »

Maybe the Claremont administration found out about this and decided to make their team look better? After all, they do cheat at college rankings.
Nicholas C
KQBA member
User avatar
Ndg
Rikku
Posts: 387
Joined: Tue Oct 30, 2012 3:24 pm
Location: Wisconsin

Re: Crowdsourced Head to Head Rankings

Post by Ndg »

The Ununtiable Twine wrote:There are so many interesting things that could come of this, say if you make everyone make an account and give some additional info to sign into this.
Yeah, this might need to be implemented for this to retain any usefulness, in light of the fact that someone has invested enough time in trolling this to give Claremont a sizable lead at #1. (And to put Maryland B at #4, four spots ahead of Maryland A.)
Andrew Nadig

Mannhiem Mannheim Manheim Township, 2005-11
Carnegie Mellon University, 2011-15
User avatar
Frater Taciturnus
Auron
Posts: 2463
Joined: Mon Dec 12, 2005 1:26 pm
Location: Richmond, VA

Re: Crowdsourced Head to Head Rankings

Post by Frater Taciturnus »

*sigh*

This... this is why we can't have nice things.
Janet Berry
[email protected]
she/they
--------------
J. Sargeant Reynolds CC 2008, 2009, 2014
Virginia Commonwealth 2010, 2011, 2012, 2013,
Douglas Freeman 2005, 2006, 2007
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: Crowdsourced Head to Head Rankings

Post by The Ununtiable Twine »

bradleykirksey wrote:
And then there's also the fact that you have people like me rating, and really, why would I know if Minneanapolis CTC is better than Alabama?
*cough 340-100 *cough - it is I who am driving Rob and Bernadette's ranking into the ground! There is simply no other explanation as to why the indefeatable Minneanapolis CTC's team could be doing so badly in the Rangaraju rankings!
Frater Taciturnus wrote:*sigh*
This... this is why we can't have nice things.
In all seriousness let's be a little more serious about seeing what kind of results we get when the PriyaRank gets reset. I think we're all curious to see how this correlates to what comes out of our normal polling methods.
Jake Sundberg
Louisiana, Alabama
retired
bradleykirksey
Wakka
Posts: 187
Joined: Sat Nov 12, 2011 5:09 pm

Re: Crowdsourced Head to Head Rankings

Post by bradleykirksey »

The Ununtiable Twine wrote:
bradleykirksey wrote: I also think that some teams are a lot better at ACF than NAQT. We played Alabama very close a few years back on NAQT tossups (losing on the last tossup of course).
That was at a combined field on D2 questions, but it was a very fun match indeed!
It stayed about a tossup apart the entire time. The only way it could have been more fun was if we got the last one of course, lol.

As for Minnesota CTC, I didn't realize that actually. I just remember them finishing one above you at ICT and not losing any starters. But since I've never seen them play, I'm really only going off of one tournament ever. Which isn't too fair. I basically have no clue what I'm talking about, and you've got people like me voting. (Though I feel like the actual Mike Cheyne polls have the same problem. We murdered WUSTL last year something like 305-100 and finished ahead of them at ICT. We rightfully got no votes in the poll but they got ranked 25th. No offense to Charles Hang, but I still don't understand why people voted for them over the only teams that finished ahead of us both.)


And Andrew and George, if UCF is ever ahead of Chicago, there's something seriously wrong. I don't think this can be a small sample size problem.

Edited because I am stupid

Edit part 2: We had a mild power spike (tied for 3rd most on the day) and relaly barely a PPB spike (less than 2.0 higher than on the day average) but we didn't neg ourselves into oblivion like usual. We only negged twice against you guys, compared to 8 times against our C team
Last edited by bradleykirksey on Tue Feb 25, 2014 7:06 am, edited 1 time in total.
Bradley Kirksey
Mayor of quiz bowl at the University of Central Florida (2010-2015)
The club at Reformed Theology Seminary Orlando (2017 - 2021)
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: Crowdsourced Head to Head Rankings

Post by AKKOLADE »

I really hope you guys aren't surprised by this.

Edit: although if someone set up a script to do it and took the site down with it, that'd be pretty annoying.
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
The Ununtiable Twine
Auron
Posts: 1058
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: Crowdsourced Head to Head Rankings

Post by The Ununtiable Twine »

Grams's Go-Go Boots wrote:I really hope you guys aren't surprised by this.

Edit: although if someone set up a script to do it and took the site down with it, that'd be pretty annoying.
I think we would have been surprised if it DIDN'T happen.
Jake Sundberg
Louisiana, Alabama
retired
bradleykirksey
Wakka
Posts: 187
Joined: Sat Nov 12, 2011 5:09 pm

Re: Crowdsourced Head to Head Rankings

Post by bradleykirksey »

Grams's Go-Go Boots wrote:I really hope you guys aren't surprised by this.

Edit: although if someone set up a script to do it and took the site down with it, that'd be pretty annoying.
Surprised isn't the word. I think disappointed is.
Bradley Kirksey
Mayor of quiz bowl at the University of Central Florida (2010-2015)
The club at Reformed Theology Seminary Orlando (2017 - 2021)
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: Crowdsourced Head to Head Rankings

Post by Rococo A Go Go »

bradleykirksey wrote:
Grams's Go-Go Boots wrote:I really hope you guys aren't surprised by this.

Edit: although if someone set up a script to do it and took the site down with it, that'd be pretty annoying.
Surprised isn't the word. I think disappointed is.
FWIW, Ram and I basically talked about trolls hitting the site as soon as he set it up, and I think this was basically just a demonstration version anyway.
Nicholas C
KQBA member
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: Crowdsourced Head to Head Rankings

Post by Rococo A Go Go »

Grams's Go-Go Boots wrote:I really hope you guys aren't surprised by this.

Edit: although if someone set up a script to do it and took the site down with it, that'd be pretty annoying.
I think those fuckers did it. Good job guys, you crashed a quizbowl alliance's website so you can have fun.
Nicholas C
KQBA member
bradleykirksey
Wakka
Posts: 187
Joined: Sat Nov 12, 2011 5:09 pm

Re: Crowdsourced Head to Head Rankings

Post by bradleykirksey »

I guess that makes me feel a little better, but I'm definitely all about Jake's suggestion of having to log in so we can keep out trolls.

When we do that, I'll be very interested by the data we get from it.
Bradley Kirksey
Mayor of quiz bowl at the University of Central Florida (2010-2015)
The club at Reformed Theology Seminary Orlando (2017 - 2021)
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

I knew as soon as I posted this that it was only a matter of time until it went to shit.
Well, now that everyone's had their fun, looks like it's time to reset the rankings and add some security. And also boot Claremont off the list.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

Okay, either this is a strange coincidence, or they took down my web hosting provider as well.

Edit: No wait, I gave them too much credit. While they did destroy the rankings, it wasn't them who took down the site. My provider had a scheduled downtime from 11:00 - 12:00 EST tonight.
Last edited by sephirothrr on Tue Feb 25, 2014 12:53 am, edited 1 time in total.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
bradleykirksey
Wakka
Posts: 187
Joined: Sat Nov 12, 2011 5:09 pm

Re: Crowdsourced Head to Head Rankings

Post by bradleykirksey »

sephirothrr wrote: And also boot Claremont off the list.
I'm assuming that that part's a joke, right? They're definitely not the worst team on the list.

Sorry though. It's frustrating when trolls break nice things.

Edit: Took down the whole provider? Forgive my stupid social-science-major question, but is that possible?

Edit x 2: Well good. I couldn't imagine the logistics of that, but I would think that would be pretty terrible.
Last edited by bradleykirksey on Tue Feb 25, 2014 12:34 am, edited 1 time in total.
Bradley Kirksey
Mayor of quiz bowl at the University of Central Florida (2010-2015)
The club at Reformed Theology Seminary Orlando (2017 - 2021)
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

bradleykirksey wrote:
sephirothrr wrote: And also boot Claremont off the list.
I'm assuming that that part's a joke, right? They're definitely not the worst team on the list.

Sorry though. It's frustrating when trolls break nice things.

Edit: Took down the whole provider? Forgive my stupid social-science-major question, but is that possible?
I'm not actually going to remove them. Probably.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

Well, since I have to wait for my database to come back online, what else would you like to see added? I've fixed the exploit that was used to screw up the ranking, and I've added Google/Facebook authentication to hopefully keep people honest. I'm a little torn on hiding rankings, because while it might help the results a little, I'm afraid that people will be less likely to participate if they can't see the effect they're having.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
bradleykirksey
Wakka
Posts: 187
Joined: Sat Nov 12, 2011 5:09 pm

Re: Crowdsourced Head to Head Rankings

Post by bradleykirksey »

sephirothrr wrote:Well, since I have to wait for my database to come back online, what else would you like to see added? I've fixed the exploit that was used to screw up the ranking, and I've added Google/Facebook authentication to hopefully keep people honest. I'm a little torn on hiding rankings, because while it might help the results a little, I'm afraid that people will be less likely to participate if they can't see the effect they're having.
My guess would be that keeping it private would be better for getting peoples' actual opinions on rankings. Keeping it public would make it a much more fun little toy. The call is yours I guess, but I think that that's what the decision boils down to.

Good news guys! Cleremont is about to get passed again! By Truman State!
Bradley Kirksey
Mayor of quiz bowl at the University of Central Florida (2010-2015)
The club at Reformed Theology Seminary Orlando (2017 - 2021)
User avatar
AKKOLADE
Sin
Posts: 15782
Joined: Thu Apr 24, 2003 8:08 am

Re: Crowdsourced Head to Head Rankings

Post by AKKOLADE »

The best apart this is it outs people who waste their time voting for themselves on these things!
Fred Morlan
University of Kentucky CoP, 2017
International Quiz Bowl Tournaments, CEO, co-owner
former PACE member, president, etc.
former hsqbrank manager, former NAQT writer & subject editor, former hsqb Administrator/Chief Administrator
User avatar
mhayes
Rikku
Posts: 308
Joined: Thu Feb 16, 2006 3:15 pm

Re: Crowdsourced Head to Head Rankings

Post by mhayes »

bradleykirksey wrote: We murdered WUSTL last year something like 305-100 and finished ahead of them at ICT. We rightfully got no votes in the poll but they got ranked 25th. No offense to Charles Hang, but I still don't understand why people voted for them over the only teams that finished ahead of us both.)
Mike instructed Top-25 voters to assume that each school had its full A-team. Richard Yu played D2 solo at the ICT, so the D1 WUSTL team was significantly weaker.
Matt Hayes

Unaffiliated
User avatar
Muriel Axon
Tidus
Posts: 729
Joined: Wed Mar 21, 2012 12:19 am

Re: Crowdsourced Head to Head Rankings

Post by Muriel Axon »

What determines the rating and the rank? Why is Texas A&M (126/423/25 currently) above Maryland B (494/281/19)?
Shan Kothari

Plymouth High School '10
Michigan State University '14
University of Minnesota '20
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

It uses the Elo system, so who you play matters as well. In this case, people have figured out how to exploit the system and choose specific matches, causing the rankings to be out of order.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
User avatar
Whiter Hydra
Auron
Posts: 1418
Joined: Tue Dec 04, 2007 8:46 pm
Location: Fairfax, VA
Contact:

Re: Crowdsourced Head to Head Rankings

Post by Whiter Hydra »

In addition, the Elo system only cares about your current rating, not how many games you've played. So if a team with a record of 100-400 were to suddenly beat a team with a record of 400-100 10 times in a row, the former's Elo will most likely shoot up past the latter's.
Harry White
TJHSST '09, Virginia Tech '13

Owner of Tournament Database Search and Quizbowl Schedule Generator
Will run stats for food
User avatar
sephirothrr
Quizbowl Detective Extraordinaire
Posts: 210
Joined: Sun Mar 23, 2008 11:00 pm
Location: Louisville, KY

Re: Crowdsourced Head to Head Rankings

Post by sephirothrr »

Also true, but in theory if users were acting in good faith it wouldn't matter.
Ramapriya
Kentucky Quizbowl Alliance
University of Louisville
duPont Manual High School
Birdofredum Sawin wrote: Fri Apr 04, 2014 1:52 pmIf you don't want to be regarded as a "raving lunatic," it might be advisable to rave less, or at least to do so in a less loony manner.
Locked