The Quizbowl Resource Center

I’m happy to announce that we’ll be hosting a truly open tournament at the University of Maryland on December 15. Open not just to all levels of human players but also to computer competitors.

https://sites.google.com/view/qanta/dec-15-2018

We’ve put on a number of previous exhibition matches, but the competition has not been truly open. Now anyone can participate. You can see some of our past videos here:

https://sites.google.com/view/qanta/past-events

However, our goal is to not just throw humans and computers against each other but to showcase the ingenuity and skill of question writers. The tournament is offering cash prizes for the best packets (you do not need to be physically present to compete in the best packet or half-packet competition). Like our expo match at PACE in June, computers will be assisting question writers in crafting quality questions that are pyramidal for both computers and humans.

You also do not need to be an AI researcher to submit a system; we are offering a baseline system that you can improve (e.g., improving data or modifying a particular subsystem).

This is the first time any competition like this has been attempted, so I’m sure there will be plenty of questions and things I’ve overlooked. I’m happy to answer any questions, and I’ll update the page as questions come up.

We've updated the page to answer questions about question distribution, preferences for teams, and fees. Copy pasted below:

Distribution (Derived from PACE)

Numbers represent half and full packet requirements.
4/8 Literature
4/8 History
5/10 Science
3/6 Fine Arts
3/6 Religion, Mythology, Philosophy, and Social Science
4/8 Current Events, Geography, and General Knowledge

Fees:

This competition is free for computer teams and free for human teams who submit a packet. If you don't submit a packet, you have to pay $50. If you only submit half a packet, you have to pay $25.

Preferences:

Because of limited equipment for Human-Computer matchups, we will have a limited field (likely a maximum of sixteen human teams). Teams will be selected (final field announcement on December 1) on a first-come, first-served basis with the following exceptions:

• Teams writing a packet will be prioritized over teams writing half packets or no packet
• Teams writing a half packet will be prioritized over teams writing no packet
• We want to make sure we have at least four HS, open, and college teams. We may prioritize later registrations from groups that aren't represented in early signups.

Jordan Brownstein, Daniel Hothem, and I are looking for teammates. Preferable those who know science.

EDIT to reflect Daniel Hothem joining our team! Looking for one more.

Given the lack of registrations for question writers and teams, we're changing the field size and fee structure.

Our goal is to bootstrap a new competition framework. If you help write questions, the tournament will be free (or better). This is not a normal tournament.
If you only submit a half packet without playing, we'll pay you $25 (you can win prizes on top of that)
If you submit a full packet without playing, we'll pay you $50 (you can win prizes on top of that)
If you submit a full packet and play as a team, we'll pay you $100 (you can win prizes on top of that)
If you submit a half packet and play as a team, we'll pay you $50 (you can win prizes on top of that)
The competition is free for computer teams (you can win prizes on top of that)

There have been several people interested in team forming. Please post here if you're looking to form a team for either writing a packet or forming a team to compete! At the moment, we currently have four teams registered, but we hope to get to at least seven.

We'll start doing weekly field updates with prods on question writing each Monday starting this Monday.

We're about a week away from our deadline for packet-writing teams (which is hopefully most teams), and we don't yet have enough teams to have a viable tournament. To help people matchmake, if you'd like to find a teammate or just want to write a packet for fun, please add your information here (separate sheets for writers and players):

https://docs.google.com/spreadsheets/d/ ... sp=sharing

As a reminder, we have hundreds of dollars of prize money available for those playing and/or writing packets.

I wouldn't mind being included on a team. My details are listed in the spreadsheet Jordan linked. I'd prefer not to write questions, but I wouldn't mind doing so if that's the only way to play.

I'm still a little iffy about being able to travel for this.

I'm not sure if I'll attend but I have most of a half packet written for this. If you are interested in contributing the science questions to get this packet finished please let me know.

Is there a list of registered teams anywhere?

Field Update (10/15/18):
Maryland A
Robert Bork
Sam Spaulding, Stephen Leh, Isaac Park
Drew Scheeler,...
Michael Bentley, Jeremy Eaton, Sam Braunfeld
Will Alston, Jordan Brownstein, Daniel Hothem
Team Dalton (NYC)

janieldensen wrote: ↑Mon Oct 08, 2018 11:10 am Field Update (10/8/18):
Maryland A
Sally Hemings
Sam Spaulding, Stephen Leh, Isaac Park
Drew Scheeler,...
Michael Bentley, Jeremy Eaton, Sam Braunfeld

Jonathan Magin will be our fourth player.

Given that this tournament will seemingly end earlier than a typical tournament, are there any side events that could be run afterwards? It's near Festivus so I guess if people have packets done a week early they could be played here, although ideally there's something more coherent available.

janieldensen wrote: ↑Mon Oct 08, 2018 11:10 am Field Update (10/8/18):
[...]
Sally Hemings
[...]

Hasn’t this team been asked to stop doing this?

Is there a length guideline for submitted questions?

Is there a length guideline for submitted questions?

Try not to go over 500 characters of questions, and unless the question is exquisitely crafted, try not to go under 400 characters. This is a really important point, thanks! We'll update the question submission interface to reflect this and also update the main page.

janieldensen wrote: ↑Mon Oct 08, 2018 11:10 am Field Update (10/8/18):
Maryland A
[Redacted]
Sam Spaulding, Stephen Leh, Isaac Park
Drew Scheeler,...
Michael Bentley, Jeremy Eaton, Sam Braunfeld
Will Alston, Jordan Brownstein, Daniel Hothem

At least one spot left for sure, but we'll take additional registrations for waitlist or possible field expansion!

ezubaric wrote: ↑Tue Oct 09, 2018 3:21 pm

Is there a length guideline for submitted questions?

Try not to go over 500 characters of questions, and unless the question is exquisitely crafted, try not to go under 400 characters. This is a really important point, thanks! We'll update the question submission interface to reflect this and also update the main page.

Is this character limit because the bots have been trained on questions of this length? As a player (and writer), I'd prefer longer questions. It's tough to write well-crafted tossups of 500 characters. The ICT proves you can definitely do this, but given that this is a guerilla tournament (unless I'm missing something), it would be great to give writers a bit more margin of error. Otherwise I predict a lot of buzzer races on cliffs. Shorter questions also lend themselves to more title drops which seem like they would benefit a computer over human players.

Mike Bentley wrote: ↑Wed Oct 10, 2018 10:25 am

ezubaric wrote: ↑Tue Oct 09, 2018 3:21 pm

Is there a length guideline for submitted questions?

Try not to go over 500 characters of questions, and unless the question is exquisitely crafted, try not to go under 400 characters. This is a really important point, thanks! We'll update the question submission interface to reflect this and also update the main page.

Is this character limit because the bots have been trained on questions of this length? As a player (and writer), I'd prefer longer questions. It's tough to write well-crafted tossups of 500 characters. The ICT proves you can definitely do this, but given that this is a guerilla tournament (unless I'm missing something), it would be great to give writers a bit more margin of error. Otherwise I predict a lot of buzzer races on cliffs. Shorter questions also lend themselves to more title drops which seem like they would benefit a computer over human players.

Mike is absolutely right that producing successful 500 character questions requires a lot of writing and editing experience. My first attempts at doing so weren't very good, and while I have no doubts that the packet submitters are better writers than my senior year self, I still suspect a guerrilla NAQT length set would be quite poor.

If we're playing D2 ICT level questions I'd like to some meaty, long early clues ala EFT (which is about the same difficulty as D2 ICT). I highly doubt D2 ICT level leadins will distinguish between, say, Jordan Brownstein and a full Maryland A team very well - you're going to get buzzer races on more than half the leadins. I get that this is meant to be a test of computers, but it's not going to be a very fun competition between the human teams and I don't think I would want to play if the 500 character limit is enforced.

Yes, we have optimized for HSNCT questions in the past, so that obviously guided my thinking. If there's a strong consensus for longer questions, I'm certainly open to that. How about 600? That's actually pretty long. (At the moment, however, the system will accept anything, so this is all open to discussion; we had foolishly not written down what we wanted.)

The other thing that I'm balancing is that I'd like to be able to get some good press coverage / reactions to this, and I don't want to go full-blown AFC Nats length questions that might turn off casual viewers too much.

600 characters I think is fine - that's about 5.5/6 lines of 10/11 Times New Roman font, respectively. If the worry is that questions will go for too long and bore viewers, I genuinely do suspect that most of these tossups will go within the first couple sentences if they're written at the advertised difficulty and played by top tier high school teams, Maryland A, or strong open teams, and that this would be the case at even a full seven lines.

To take one example: let's look at the top bracket of the 2017 EFT site at the University of Virginia. I think it's safe to say that Penn A, Johns Hopkins, Maryland A, and Duke are fairly representative of the strength of our field (possibly an underestimate, considering that there are open teams and computers) and EFT 2017 is fairly representative of the target difficulty. These four teams played a total of nine games against each other. Seven of these games saw more 15s than 10s; two saw equal numbers of 15s and 10s; not a single saw fewer 15s than 10s. This doesn't even take into account the fact that negs during these games are depressing the power counts, and are likely to be fairly early in a game between two strong teams. With anywhere between 50 and 75% of buzzes occurring in the first few sentences, I doubt the questions will be sitting around for an audience for too long; furthermore, compressing questions down (likely by removing leadins and middle clues) is going to generate a ridiculous number of buzzer races.

I think a good compromise might be "maximum six lines of ten-point Times New Roman" which comes out to an effective limit of about 650 characters. That would also be very easy for writers to implement, as they wouldn't have to do a character count for each question, and would also be easy for the tournament organizers to enforce without doing the same.

Periplus of the Erythraean Sea wrote: ↑Wed Oct 10, 2018 5:16 pm 600 characters I think is fine - that's about 5.5/6 lines of 10/11 Times New Roman font, respectively. If the worry is that questions will go for too long and bore viewers, I genuinely do suspect that most of these tossups will go within the first couple sentences if they're written at the advertised difficulty and played by top tier high school teams, Maryland A, or strong open teams, and that this would be the case at even a full seven lines.

To take one example: let's look at the top bracket of the 2017 EFT site at the University of Virginia. I think it's safe to say that Penn A, Johns Hopkins, Maryland A, and Duke are fairly representative of the strength of our field (possibly an underestimate, considering that there are open teams and computers) and EFT 2017 is fairly representative of the target difficulty. These four teams played a total of nine games against each other. Seven of these games saw more 15s than 10s; two saw equal numbers of 15s and 10s; not a single had fewer 15s than 10s. This doesn't even take into account the fact that negs during these games are depressing the power counts, and are likely to be fairly early in a game between two strong teams. With anywhere between 50 and 75% of buzzes occurring in the first few sentences, I doubt the questions will be sitting around for an audience for too long; furthermore, compressing questions down (likely by removing leadins and middle clues) is going to generate a ridiculous number of buzzer races.

I think a good compromise might be "maximum six lines of ten-point Times New Roman" which comes out to an effective limit of about 650 characters. That would also be very easy for writers to implement, as they wouldn't have to do a character count for each question, and would also be easy for the tournament organizers to enforce without doing the same.

I think this is a reasonable proposal.

I should add one thing: I realize that you need to submit questions through the write.qanta.org site - however, in my experience teams usually compile submitted packets on something like Google Docs before they submit them, to help figure out who's writing what, etc. In this context, a visually simple cap is much easier for players to work with.

Okay, 600 it is. I realize that people use Google Docs, but "lines" is arbitrary, and it's not just that questions need to be submitted on qanta.org, but if you want to do well in the question writing competition, you'll need to do your final editing there (e.g., to make sure that you haven't included a computer giveaway in the first line). And a visual constraint will not be enforceable there because it depends on browser settings.

(To get on a hobby horse of mine when I interact with publishers, providing guidelines based on rendering in WYSWIG is incredibly unfair to blind people. I had a blind student who had to ask someone if he met a silly length guideline because he was unable to to it himself. Character constraints are both more accessible and precise.)

I understand that Maryland A vs. Brownstein will have relatively snappy pacing, but not all games will be like that, and I fully expect that some computer games may have a lot of negs, causing questions to be read in their entirety. So I want to still have good pacing even in the worse case, but I think 600 is a decent compromise for everyone.

Thanks for the suggestions! Keep 'em coming!

Nice hockey Cote d'Azur wrote: ↑Tue Oct 09, 2018 11:36 am

janieldensen wrote: ↑Mon Oct 08, 2018 11:10 am Field Update (10/8/18):
[...]
Sally Hemings
[...]

Hasn’t this team been asked to stop doing this?

They have.

Seriously, I think I and others understand that pseudonyms are often necessary, but this specific pseudonym needs to stop being a thing, and TDs need to stop allowing this team to register under this pseudonym. They can play as "Monticello", or "Daveed Diggs", or any number of other names which will make their actual affiliation more or less obvious without being inappropriate.

We agree with the problematic name and have been in communication with the team, all future references to the team will use Robert Bork.

Some questions that we've gotten on submitting questions, added to the FAQ on the webpage:

Q: Can we blend subcategories/categories?

Feel free to blend subcategories and categories. Our distribution requirements aren't super strict. But don't use blending to avoid writing about topics (e.g., "real" science) that should be covered in the set somewhere.

Q: Should the computer be able to answer all of the questions by the end?

No, while your answers have to be in our answer set, if the system on write.qanta.org cannot answer the question, that's fine too! It probably means that you're doing something unique and interesting. If humans will like and convert the question, then you're doing everything exactly right.

We had some complaints from people considering writing packets (without playing) that the payment for just writing a packet (50 TU) or half packet (25 TU) should be more than that of a team that also plays. We agree with that logic, and have upped the payment for half and full packets to $75 and $150 (and up to $350 more if you win the writing competition). If you're interested in writing a packet, please register by Monday!

Human field update:

* Roger Bork TBA
* Quizbowl: A "Bag of Nerds" Approach to Named Entity Recognition Sam Spaulding, Stephen Leh, Isaac Park
* still working on a name Drew Scheeler + TBA
* Maryland A Graham Reid, Caleb Kendrick, Weijia Cheng, Justin Hawkins
* What to Do After You Hit Return Michael Bentley, Jeremy Eaton, Sam Braunfeld, TBA
* Rage Against the Machines Will Alston, Jordan Brownstein, Daniel Hothem
* Team Dalton (NYC) Avi Mehta

We're extending the deadline to register for writing packets until the end of the week, so please sign up if you're interested in contributing a half packet ($75) or full packet ($150).

Are the packets really going to be 50 tossups?

No, we'll edit down to 40 + tiebreakers.

Had some good questions come through. Updated the webpage:

https://sites.google.com/view/qanta/dec-15-2018

and also posting fuller answers here.

Cheynem wrote: ↑Tue Oct 16, 2018 10:52 am Are the packets really going to be 50 tossups?

Q: What's the format of the games? Why is it structured like that?

We'll have 40 questions, tossups only. If there's a tie after 40 questions, we'll break the tie with three tie breaker questions. The team with the higher score after three questions will win. If the score is still tied after the tie-breaker questions, we'll read questions until the score changes. The first change in the score will decide the game.
We are not using bonus questions because tossup questions are more interesting (based on their pyramidal structure) to decide whether humans or computers are smarter. What would make bonus questions interesting would be to emphasize collaboration in human-computer hybrid teams. We're working on figuring out how to make that both interesting and fun, but we're not quite there yet.

I see the question writing guidelines says we're limited to answers that are Wikipedia article titles but have noticed the QANTA question submission page only allows certain answers. Are we limited to the online list of answers? And if we are limited to the available answers is there a way to get access to that acceptable answer set?

All questions must be submitted through write.qanta.org through a single account. You can either share passwords or have a single person submit all of the questions. The goal is to make the first 1-2 lines of the questions unanswerable for computers but otherwise leave the questions something that humans would enjoy (you can see an example in the video on the webpage).

As you write questions, you'll be asked to specify not just the answer but also the category of the questions. As you write the questions required for the packet, the count will keep track of the questions you've written and let you know when you've finished your packet. Specifying the answer to the question is important because it allows us to have a fair way to adjudicate whether computers have given the correct answer or not. Only things that have Wikipedia pages can be answers. The interface makes it easy to find the right page title, though; just start typing, and it will autocomplete with the possible page titles.

We're getting more great questions ... following the usual strategy of posting on the webpage and the message boards.

Q: I can't find my favorite answer in the system. Why is this?

This is a design decision that we made. These are the answer for which there have been three answers in mainstream quiz bowl tournaments. This is a tradeoff that we made to keep things relatively fair. We want questions that are challenging for computers not because they lack data but because they cannot understand English. By excluding rare answers and only focusing on frequently asked answers, if a computer gets it wrong, it's not because it lacks information to work off of ... it's because it didn't understand the question. We realize it's a little frustrating, so it's useful to check whether the answer is in bounds before writing the question. In many cases, you can tweak the question to ask about something more general (instead of asking about "William W. Belknap", ask about Grant, focusing on members of his cabinet).

ezubaric wrote: Only things that have Wikipedia pages can be answers.

I think some people (myself included) interpreted this to mean "if X is a Wikipedia page then X can be an answerline," not "the set of submittable answerlines is a subset of Wikipedia article titles." Based on the below post the restriction is the latter.

ezubaric wrote: ↑Wed Oct 17, 2018 2:35 pm We're getting more great questions ... following the usual strategy of posting on the webpage and the message boards.

Q: I can't find my favorite answer in the system. Why is this?

This is a design decision that we made. These are the answer for which there have been three answers in mainstream quiz bowl tournaments. This is a tradeoff that we made to keep things relatively fair. We want questions that are challenging for computers not because they lack data but because they cannot understand English. By excluding rare answers and only focusing on frequently asked answers, if a computer gets it wrong, it's not because it lacks information to work off of ... it's because it didn't understand the question. We realize it's a little frustrating, so it's useful to check whether the answer is in bounds before writing the question. In many cases, you can tweak the question to ask about something more general (instead of asking about "William W. Belknap", ask about Grant, focusing on members of his cabinet).

While I understand the motivation to have answerlines that have come up before, but I'm a little confused by the situation. Is the write.qanta site supposed to restrict your answer to things that have Wikipedia articles and have been asked about in at least three archived questions? At the moment there are numerous answers you can put into write.qanta that have not appeared as answers three times before. (Although not every Wikipedia article is an allowed answers.)

Again I respect the desire to make these submissions useful for AI research, but these restrictions might make things difficult, since teams have probably already started the writing process, and this restriction wasn't exactly clear earlier. What if someone wrote a question that doesn't meet the criteria before they were announced? What if someone used an answer that write.qanta accepted that has never been an answer for a past question (or has been an answer for fewer than three past questions and thus doesn't meet the criteria)?

a bird wrote: ↑Wed Oct 17, 2018 4:46 pm

ezubaric wrote: Only things that have Wikipedia pages can be answers.

I think some people (myself included) interpreted this to mean "if X is a Wikipedia page then X can be an answerline," not "the set of submittable answerlines is a subset of Wikipedia article titles." Based on the below post the restriction is the latter.

ezubaric wrote: ↑Wed Oct 17, 2018 2:35 pm We're getting more great questions ... following the usual strategy of posting on the webpage and the message boards.

Q: I can't find my favorite answer in the system. Why is this?

This is a design decision that we made. These are the answer for which there have been three answers in mainstream quiz bowl tournaments. This is a tradeoff that we made to keep things relatively fair. We want questions that are challenging for computers not because they lack data but because they cannot understand English. By excluding rare answers and only focusing on frequently asked answers, if a computer gets it wrong, it's not because it lacks information to work off of ... it's because it didn't understand the question. We realize it's a little frustrating, so it's useful to check whether the answer is in bounds before writing the question. In many cases, you can tweak the question to ask about something more general (instead of asking about "William W. Belknap", ask about Grant, focusing on members of his cabinet).

While I understand the motivation to have answerlines that have come up before, but I'm a little confused by the situation. Is the write.qanta site supposed to restrict your answer to things that have Wikipedia articles and have been asked about in at least three archived questions? At the moment there are numerous answers you can put into write.qanta that have not appeared as answers three times before. (Although not every Wikipedia article is an allowed answers.)

Again I respect the desire to make these submissions useful for AI research, but these restrictions might make things difficult, since teams have probably already started the writing process, and this restriction wasn't exactly clear earlier. What if someone wrote a question that doesn't meet the criteria before they were announced? What if someone used an answer that write.qanta accepted that has never been an answer for a past question (or has been an answer for fewer than three past questions and thus doesn't meet the criteria)?

Yeah I think I'm likely in this situation but haven't confirmed yet.

a bird wrote: ↑Wed Oct 17, 2018 4:46 pmWhile I understand the motivation to have answerlines that have come up before, but I'm a little confused by the situation. Is the write.qanta site supposed to restrict your answer to things that have Wikipedia articles and have been asked about in at least three archived questions? At the moment there are numerous answers you can put into write.qanta that have not appeared as answers three times before. (Although not every Wikipedia article is an allowed answers.)

Again I respect the desire to make these submissions useful for AI research, but these restrictions might make things difficult, since teams have probably already started the writing process, and this restriction wasn't exactly clear earlier. What if someone wrote a question that doesn't meet the criteria before they were announced? What if someone used an answer that write.qanta accepted that has never been an answer for a past question (or has been an answer for fewer than three past questions and thus doesn't meet the criteria)?

First, we were indeed not shouting this from the rooftops as loudly as we could. We were telling people to submit to write.qanta.org, but not doing a good job of explaining why. We'll redouble our efforts to explain this.

Yes, the site restricts answers to things that have Wikipedia page titles and have been asked about three times. So if you can select it as an answer, it's totally fine! (We may have more data than the DBs you're using ... we also have lower standards.)

I understand how this might be an inconvenience, and we're happy to work with people (off the boards, obvious, for the sake of question security). If you have a lot of answerlines that are out of bounds, talk to us directly. If they're Wikipedia pages that just haven't been asked about before, we can likely just flag that as a special case (it's extra work for our editors). If it's not a Wikipedia page, then we can have a longer discussion. As you said, this is about research, so we can learn from each other!

ezubaric wrote: ↑Wed Oct 17, 2018 5:19 pm Yes, the site restricts answers to things that have Wikipedia page titles and have been asked about three times. So if you can select it as an answer, it's totally fine! (We may have more data than the DBs you're using ... we also have lower standards.)

Maybe I'm missing something, but I still think the submission system isn't restricting answers properly. There are lots of allowed answers for which the "questions about_" box is blank. Are there questions in the dataset that don't appear in this tab? I've also found several allowed answers like "List of Bulgarian monarchs" that are Wikipedia articles but have (presumably) never been an answer in quizbowl (at least not in that exact form).

Yes, that's right. It's probably not worth getting into details, but there are multiple signals that we can use to generate whether something has been asked about: Jeopardy!, other trivia questions online, etc. We don't think showing those questions are worth showing to quiz bowl writers, though.

The list pages specifically are used for common link questions.

Here's a question about the write.qanta interface that I don't think I can figure out without going in and trying it: if I start writing a question, at what point is it submitted to the website? That is, if I start writing a question with the answer "John Doe", am I locked into having a John Doe question in my packet, or can I later decide that my John Doe ideas aren't working out and change it to a question on "Mary Smith"? Relatedly, if I write more than 40 questions, can I somehow flag which ones I want to officially submit?

Louis XIV and Twenty Million Henchmen wrote: ↑Wed Oct 17, 2018 9:00 pm Here's a question about the write.qanta interface that I don't think I can figure out without going in and trying it: if I start writing a question, at what point is it submitted to the website? That is, if I start writing a question with the answer "John Doe", am I locked into having a John Doe question in my packet, or can I later decide that my John Doe ideas aren't working out and change it to a question on "Mary Smith"? Relatedly, if I write more than 40 questions, can I somehow flag which ones I want to officially submit?

A great question!

It only gets saved on the server when you hit submit. So if you work on a question for a while and realize that it just isn't going to work, you don't have to do anything.

Once you hit Submit, you can go back to edit as much as you want.

But ... this now makes me realize that we don't have a way to delete questions. I'll pass that along to Eric.

Human Field Update
* Roger Bork TBA
* Quizbowl: A "Bag of Nerds" Approach to Named Entity Recognition: Sam Spaulding, Stephen Leh, Isaac Park
* Still working on a name: Drew Scheeler + TBA
* Maryland A: Graham Reid, Caleb Kendrick, Weijia Cheng, Justin Hawkins
* What to Do After You Hit Return: Michael Bentley, Jeremy Eaton, Sam Braunfeld, TBA
* Rage Against the Machines: Will Alston, Jordan Brownstein, Daniel Hothem
* Team Dalton (NYC): Avi Mehta
* Miriam Nussbaum (Only writing)

Also, we have updated our web page, https://sites.google.com/view/qanta/dec-15-2018, with more information about writing questions.

Q:What if I already have some questions written that aren’t in the interface?

Due to the limitations of this competition, it will be necessary to restrict answer lines to the ones already in the answer set at write.qanta.org. It isn’t recommended, but if you have already written questions with answers outside of the set, you can email them to [email protected] to submit them. Otherwise, I would recommend that you check if an answer is available at write.qanta.org before writing a question.


Q:What is the best way to write questions

We suggest the following procedure for writing questions. First, as you’re deciding on answer lines, check write.qanta.org to make sure they’re in bounds. Then, draft the question in whatever editor you’re most comfortable with. When you have a first draft, copy and paste it into the interface, edit to make sure the lead in (at least) isn’t trivially answerable by our baseline system, and then submit. We also suggest keeping your own backup just in case something goes wrong (we hope it doesn’t, but better safe than sorry). We realize this is slightly more hassle than normal question writing, but this will hopefully lead to better questions and also advance the state of the art in natural language processing.

Two brief updates. The framework for computer teams to submit systems is now online:

https://pinafore.github.io/qanta-leaderboard/

We're still accepting human registrations for the waiting list. If we get to 14 teams, we'll likely expand the field. Preference will be given to those writing questions.

We've extended the deadline for registering for the computer half of the competition to November 28. We currently have 16 computer teams registered:

* SFU NLP
* Simplifai
* ugrad
* Studio OUSIA
* Clever Name TBD
* FYY-723
* 723 WT
* Z - NLP
* FFT
* Iota
* QaQa Land
* Forward Thinking
* CALEZH
* Epsilon
* AmazingDH

We also wanted to clarify the format for the computer round robin. Each computer team will play on a "packet" of the same 280 questions against each other computer competitor (this is possible because we can erase the computer team's memory and have it replay the game against a new opponent). These are all of the same questions the humans will hear in their non-amnesia round robin in the morning. The teams with the most wins will advance to the human-computer playoffs (ties broken by PPG).

We just got some new interest from some high school teams ... we could expand the field if we just had one or two more teams.

No registration fee, no packet writing requirements ... just show up! (First come first served.)

If you're interested, sign up on http://qanta.org

Please join us at the University of Maryland for our first Human-Computer Question Answering competition. Our event will be in the Language Science Center:
https://languagescience.umd.edu/about/directions

Please come to HJP 2130 between 8:15 and 8:45 for registration and some breakfast foods. We’ll have a brief rules meeting and explanation of what’s going on at 8:45. Games start at 9:00 sharp.

Congrats to the teams that competed in our tournament today!

Computer Division: FYY
High School: Monticello
College: Maryland A
Open: Rage Against the Machine

Full prize information, stats, and videos after I get grades in. In the meantime, please don't discuss questions as we're going to have an online mirror.

Thanks to Jordan B-G and co. for putting on an excellent event, and to Jordan, JinAh, and Daniel for being great teammates. More tournaments should offer Krazy Kebab catering for lunch.

To speak a bit about the questions and the format: If you get a chance to play this question set online, I recommend doing so - for a set that was largely guerilla, it was pretty good and had a number of interesting questions, and I think it gradated between the teams who actually played the tournament decently well. Playing 40-tossup games definitely helped remove the variance associated with tossup-only tournaments as well.

ezubaric wrote: ↑Sat Dec 15, 2018 10:25 pmFull prize information, stats, and videos after I get grades in. In the meantime, please don't discuss questions as we're going to have an online mirror.

Thanks for running this, it was a blast. Will videos and full results be posted soon?

Sorry for the delay. My first priority was getting checks out to people, and we have mostly complete stats here:

http://users.umiacs.umd.edu/~jbg/qb/201 ... games.html

I know there is one missing game of Monticello vs. Technical Wizards, which I'll have to recreate from the video. Video is underway, but I want to create all of the videos before I start posting anything so it may be a while yet. Also trying to work out our question security issues for some additional experiments and online mirrors, so don't want to post videos/questions before then.

ezubaric wrote: ↑Sun Jan 06, 2019 10:37 pm Sorry for the delay. My first priority was getting checks out to people, and we have mostly complete stats here:

http://users.umiacs.umd.edu/~jbg/qb/201 ... games.html

I know there is one missing game of Monticello vs. Technical Wizards, which I'll have to recreate from the video. Video is underway, but I want to create all of the videos before I start posting anything so it may be a while yet. Also trying to work out our question security issues for some additional experiments and online mirrors, so don't want to post videos/questions before then.

What's the plans with regards to posting packets from this tournament?

The were used on QBlitz, but now that's over we should be posting packets. The JSON versions are coming soon, but in the meantime, we have Google Docs for the impatient.

http://trickme.qanta.org

Videos / press release are hopefully coming in the next couple of weeks along with our publication announcement.