Some new question database

[Eärendil]

It can be found here.

I had no plans for the summer and I thought I'd learn some computer programming, so I decided to take on the idea of a question database. I know it's been done before, but I wanted to address three particular issues that intrigued me:

Full Unicode support*: This means that any and all diacritical or special characters that were originally in the packet are preserved. This might not seem like a big deal, but I've always appreciated these emebellishments. What's more, you don't need to specify diacritics in your search; just typing in "bela bartok" will catch instances of "béla bartók" as well.

A large source of questions: For college tournaments, this means ACF and many mACF tournaments. Not all of them, but at least the ones that were generally well-received. In addition, there's plenty of freely-available high school tournaments as well, so I can't see why they shouldn't be a part of this as well. As of now, PACE and other house-written sets are up. Trash tournaments are part of this too, since there's plenty of well-written and interesting trash questions out there. Currently, only tournaments from 2008 have been added to the database, though I plan to add older tournaments as well.

Very flexible search options: By default, searches will only look through collegiate, academic tournaments, as I feel this will be the most popular source of questions people would want to look through. However, this can be easily changed under "Search Options", as well as looking at only answer choices vs. clues, tossups vs. bonuses, or limiting results by year. Under "Subject Options", you can choose which subjects to include and whether to have them included with an "and" or an "or". It's the difference between retrieving questions tagged with "American Literature" and "Poetry", or questions tagged with "American Literature" or "Poetry". To retrieve all questions within a certain subject, select the appropriate checkboxes and leave the search field empty. If no subjects are selected, all subjects will be searched; this combined with an empty search field, however, will not retrieve all the questions in the database.

Subject tagging in particular has been made extremely accessible. Anyone can categorize a question as they see fit. Moreover, if a question is miscategorized, it can be fixed instantly, instead of having the webmaster see to it. I'm leaving this feature of the site open, for now. Hopefully there won't be any abuse of the system (e.g. evolution tossups being categorized as religion).

There's a bit more to the site, but I'd rather you try it out. Let me know what you guys think of it.

* Technically untrue, as MySQL doesn't support all Unicode tables, but as long as quiz bowlers don't start writing Amaterasu in hiragana, the current scheme should cover pretty much everything that comes up.

[Gautam]

I loooooove the title!

[Frater Taciturnus]

* [Technically untrue, as MySQL doesn't support all Unicode tables, but as long as quiz bowlers don't start writing Amaterasu in hiragana, the current scheme should cover pretty much everything that comes up.

Damn

[Maxwell Sniffingwell]

* [Technically untrue, as MySQL doesn't support all Unicode tables, but as long as quiz bowlers don't start writing Amaterasu in hiragana, the current scheme should cover pretty much everything that comes up.

ANSWER: Six Records of a Floating Life [accept 大中华文库—浮生六记]

I've always wondered about that one.

[grapesmoker]

How are you doing your importing?

[Eärendil]

I first convert everything (docs, PDFs, RTFs, etc.) to a UTF-8 text file. Then I clean up some commonly-found patterns with regular expression matching. At this point, I look over the text file and manually make edits to clean up the questions, so they're easier to detect as either a tossup or a bonus. Then, regular expressions match tossups, categorize them as tossups, and insert them into the database. Repeat, except with bonuses.

My biggest complaint so far has been inconsistent bonus stylings. For what it's worth, I think that the point value of every part should be at the beginning of the clue, enclosed in [ ], as in the following example:

23. He was the first Congressman to serve for over 40 years, but his reputation was diminished by his pugnacious manner. For the stated points:
[10] For 10, identify this ‘explosive’ speaker of the House of Representatives, who held the post from 1903-1911.
ANSWER: Joseph Gurney Cannnon
[10] For 10, Cannon used his power to help pass this 1909 tariff, which may have been the key to President Taft’s undoing in the 1912 election.
ANSWER: Payne-Aldrich Tariff
[5]/[5] For 5 points each, Taft was also hurt by the controversy between these two men: one the chief of the Forest Service, the other a Secretary of the Interior who opened public lands in Alaska for private development.
ANSWER: Gifford Pinchot and Richard Ballinger [either order, 5 points for each correct answer]

I hate having to deal with "A. ... ANSWER: ... B. ... ANSWER: ... C. ... ANSWER. ..." or some other scheme. For the algorithm, these look too much like tossups and screws everything up.

Also, I forgot to mention this earlier, but since this might be of interest to high schoolers and trash enthusiasts, could the mods please cross-post this appropriately? Much thanks.

EDIT: I can't spell.

[leapfrog314]

I decided to take on the idea of a question database. I know it's been done before,

This looks cool, and I like that you've stressed the Unicode thing here. But pretty soon we're going to have to declare a moratorium on new question databases!

[Auks Ran Ova]

I decided to take on the idea of a question database. I know it's been done before,

This looks cool, and I like that you've stressed the Unicode thing here. But pretty soon we're going to have to declare a moratorium on new question databases!

Obviously the best solution is to create some sort of searchable database for them.

[grapesmoker]

I think the issues involved in parsing packets are common to all the various attempts at having a database. Here's my proposal: the CS-minded among us should pool our resources and work on one database. It doesn't have to be mine, but I think it should be similar in design, with other capabilities built in (categories, quality rankings, difficulty, etc.).

[Skepticism and Animal Feed]

I think ACFDB and Gyaankosh are fine; I haven't encountered a single bug while using them. Is the hard part just re-formatting packets so they can be used? If quizbowl players don't have time to do that, could we just hire some company (perhaps in the third world) to do it for us? I'd be willing to donate a portion of my proceeds from RMPfest and other stuff I might write in the future to do it (and I'm sure other writers and/or individuals would too) because I just find the databases so useful.

[grapesmoker]

I think ACFDB and Gyaankosh are fine; I haven't encountered a single bug while using them. Is the hard part just re-formatting packets so they can be used? If quizbowl players don't have time to do that, could we just hire some company (perhaps in the third world) to do it for us? I'd be willing to donate a portion of my proceeds from RMPfest and other stuff I might write in the future to do it (and I'm sure other writers and/or individuals would too) because I just find the databases so useful.

Format your packets properly to begin with and we won't have this problem.

[Skepticism and Animal Feed]

Right, so a different solution to the problem would be a time machine to allow Jerry to yell at people who wrote packets before the modern formatting standards were adopted (which was what, last year?).

[grapesmoker]

Right, so a different solution to the problem would be a time machine to allow Jerry to yell at people who wrote packets before the modern formatting standards were adopted (which was what, last year?).

Bruce, do you have anything to contribute to this discussion or are you just being contrarian?

[Skepticism and Animal Feed]

The question I asked was whether or not the problem with having a comprehensive database is just an issue of having to re-format packets. You responded by telling people to properly format their packets. I pointed out that I was referring to old packets whose authors did not know the current standard.

I suspect that some people here don't actually read my posts, they just say "ah, I've disagreed with this guy in the past, therefore he must be saying something terrible right here."

[grapesmoker]

The question I asked was whether or not the problem with having a comprehensive database is just an issue of having to re-format packets. You responded by telling people to properly format their packets. I pointed out that I was referring to old packets whose authors did not know the current standard.

I suspect that some people here don't actually read my posts, they just say "ah, I've disagreed with this guy in the past, therefore he must be saying something terrible right here."

Yes, the issue is formatting the packets. This has been obvious from day one and anyone who's taken a crack at a question database understands this. Old packets will have to be reformatted to current standards if we want them to be entered into a searchable database. However, the notion of outsourcing this job is laughable; instead, what needs to happen is that a bunch of people take on the distributed task of formatting these packets.

[Skepticism and Animal Feed]

Pretty much everyone in quizbowl has an obvious interest in those packets getting formatted and submitted to a database. The original thread from QBDB has dozens of people volunteering to format packets in their spare time. These seem to be just as on track as the Greg Peterson MO packet.

There's some kind of failure here, and I'm wondering what needs to happen before people actually start reformatting packets. Does the community have to settle on a single database? Do people need to be given financial incentives? Etc.

[No Rules Westbrook]

I suspect that some people here don't actually read my posts, they just say "ah, I've disagreed with this guy in the past, therefore he must be saying something terrible right here."

Been there, brother speaks the truth.

But, anyways, comprehensive question databases (searchable or even non-searchable) are just grand. I think they may be the single most important development toward the goal of making new entrants into good college players and writers.

[grapesmoker]

Pretty much everyone in quizbowl has an obvious interest in those packets getting formatted and submitted to a database. The original thread from QBDB has dozens of people volunteering to format packets in their spare time. These seem to be just as on track as the Greg Peterson MO packet.

There's some kind of failure here, and I'm wondering what needs to happen before people actually start reformatting packets. Does the community have to settle on a single database? Do people need to be given financial incentives? Etc.

Hey, how about this for consideration:

I'm really fucking busy, all the goddamn time. I just finished writing a quarter of a tournament, and I'm about to start putting together ACF Winter; before that, I was living outside of my home for more than half a year because I had to work on my project, which I am currently in the process of quitting; which means that I have to find a new advisor to work with and I still have a ton of loose ends to wrap up before I finally leave this group in January (by which time I'll be on the hook for ACF Regionals and Penn Bowl packets).

I'm sorry that my peripatetic lifestyle and the myriad of things I have to do in order to live and support myself and make progress towards my degree are getting in the way of working on something that no one is obligated to provide. If I had a couple months off from my work, I could probably do many of these things, but I don't have a couple months. As a result, things that are lower in priority tend to fall by the wayside and things that are higher in priority, like, you know, producing tournaments and stuff get done first. But at least, unlike you, I have taken a non-negligible amount of time and effort to advance a project I would like to see completed; unlike you, I don't come into these threads with rhetorical questions about what can be done and bemoaning the alleged failure of an undertaking towards which I've contributed nothing.

You don't have to be a genius to understand what needs to happen for this to work: someone needs to sit down and parcel out the workload to the various people who are interested in contributing. Arguably I should have done that, but I didn't. If you want to help by taking on this task, then be my guest, but this bullshit about financial incentives (what financial incentives?) and musings about how many databases we need is completely unhelpful.

[Maxwell Sniffingwell]

These seem to be just as on track as the Greg Peterson MO packet.

?

[BuzzerZen]

These seem to be just as on track as the Greg Peterson MO packet.

?

See, it's a joke at your expense.

[Mechanical Beasts]

These seem to be just as on track as the Greg Peterson MO packet.

?

The meaning here is "not."

Jerry, I don't think that Bruce is trying to be inflammatory, and I don't think that he means to insinuate that you, yourself, are personally at fault for not taking control of this operation. He wants to get this thing done, and wants to know what it would take to get people in charge of assigning work and then doing that work. Moreover, he wants to suppose that maybe financial incentives--which obviously haven't been tried or even really considered--could be helpful.

[No Rules Westbrook]

Yeah, plus, let's remember that Bruce views all individuals as POLITIES which interact with each other according to Realist norms, so it's only natural that he'd suggest something like financial incentive.

[Matt Weiner]

Hey, an easy way to simplify the task of reformatting old packets is to write a parser that strips out everything besides text, including paragraph returns and such, and replaces it with a space. Then, the human labor only has to go through and insert returns between blocks of question text, answer lines, the next question, etc. That seems like it would be much easier than asking people to get rid of all the indents and stuff on their own.

[grapesmoker]

Jerry, I don't think that Bruce is trying to be inflammatory, and I don't think that he means to insinuate that you, yourself, are personally at fault for not taking control of this operation. He wants to get this thing done, and wants to know what it would take to get people in charge of assigning work and then doing that work. Moreover, he wants to suppose that maybe financial incentives--which obviously haven't been tried or even really considered--could be helpful.

It's standard Bruce operating procedure to suggest unworkable solutions to nonexistent problems and then get defensive when people point out these simple facts. Whatever, this thread is about neither my feelings nor Bruce. I will repeat again that the notion of financial incentives is absurd on its face; we can barely get people to properly format their packets correctly and we already offer financial incentives for that. Not to mention that no on in quizbowl has any money, so what's going to be funding these incentives is unclear.

I'll take a step forward in this: if you are interested in adding to QBDB's content, you can help out by formatting packets. I've set up a spreadsheet on Google Docs to keep track of the task of formatting packets. Basically, the way it works is like this: if you want to help, send me an email at [email protected] with "[qbdb]" in the subject line and I will invite you to collaborate on the spreadsheet. Then, grab a tournament off the Stanford archive, format it according to the ACF guidelines (including renaming the packets with the correct scheme), and send me a zip file with the set. When you decide to work a tournament, enter the information into the columns provided so that no one else tries to do the same thing. When I've uploaded the set, I will note that in the spreadsheet and update the thread.

Last time I tried soliciting help, I had people send me all sorts of packets which were formatted wrong. I hope it doesn't happen again, and I'm glad to post the guidelines again if people need to see them. As I've said before, if you want to see this happen, pitch in. It's too much work for me to handle as one person, but if a bunch of people take an hour out of their week to format a single set, we're going to have this done very fast.

I certainly don't want to undermine anyone else's efforts to create a packet database. I do think that my program is farther along than the others just because I've worked on it longer, and I've gotten most of the bugs out (although I still haven't figured out how to handle Unicode because I'm using XML as an intermediary; suggestions are welcome). Obviously, features like categories and difficulty/quality rankings would be great too, and I do plan to add them, but the infrastructure for a useful database is there and people could make use of it effectively if more data was entered.

[Eärendil]

Hey, an easy way to simplify the task of reformatting old packets is to write a parser that strips out everything besides text, including paragraph returns and such, and replaces it with a space. Then, the human labor only has to go through and insert returns between blocks of question text, answer lines, the next question, etc. That seems like it would be much easier than asking people to get rid of all the indents and stuff on their own.

This is almost exactly the way my parser works, except that it saves time and inserts line feeds whenever a packet is formatted correctly. The worst are PDFs, which usually come out as a continuous block of text and need to be manually scanned through, but it's still not abominable.

Obviously, features like [...] difficulty/quality rankings would be great too

I had thought of putting in a difficulty/quality ranking system into Gyaankosh, but scrapped it when I realized it probably wouldn't be ready by early September. I'll look into it as a future addition. The other major implementation I'd like to put in is preservation of italics, bolds, and underlines, but that I don't think I'll have time to introduce anything new until January. That said, people do seem to find this useful, so I'll continue to add older tournaments. On the other hand, as far as usability is concerned, what other features would be useful? Once enough questions have been placed into subjects, we can work out a simple frequency list or concordance-like feature, but I'm interested in hearing what else would be worthwhile.

Does the community have to settle on a single database?

I don't think so. I think Carlo's, Jerry's, and my databases have slightly different ideological approaches and different functionalities, which some people may like and others may not. It also helps to have multiple redundancies, in case one of the goes down or something.

Here's my proposal: the CS-minded among us should pool our resources and work on one database. It doesn't have to be mine, but I think it should be similar in design, with other capabilities built in (categories, quality rankings, difficulty, etc.).

Would people be interested in this? I'd be glad to offer my database as a starting point, but I think we'll run into problems with different programming experiences. Personally, I'm very comfortable with Python, but I have very little knowledge of PHP, which I suspect many people are going to use as their scripting language of choice. On the other hand, I do know that I wouldn't mind having another admin or two to help me run Gyaankosh and import packets, if people are interested.

[Blackboard Monitor Vimes]

A database that preserves underlining would be awesome. I've yet to come across one, and I'm frequently curious as to how much of some titles are underlined.

[grapesmoker]

A database that preserves underlining would be awesome. I've yet to come across one, and I'm frequently curious as to how much of some titles are underlined.

I'm pretty sure my import scheme preserves underlining. In fact, it certainly does, since you can see it in the answer formatting. It's just that most titles in the question text are either italicized or in quotes.

[Blackboard Monitor Vimes]

A database that preserves underlining would be awesome. I've yet to come across one, and I'm frequently curious as to how much of some titles are underlined.

I'm pretty sure my import scheme preserves underlining. In fact, it certainly does, since you can see it in the answer formatting. It's just that most titles in the question text are either italicized or in quotes.

I had forgotten about that, actually, as I haven't used your database since early last summer. Now that school's started and I have no time for anything, I've directed most of my attention to stuff I can download to my laptop for use at practice, as I can't get internet at school, but a quick glance at what we read Friday reveals that that has underlining as well. I think it might actually just have been Carlo's that I was thinking of, actually. My memory notably fails at life.

[AlphaQuizBowler]

Is there a reason why I can't categorize tossups? The uncategorized tossups are there, and you can click add subject, but when you do they remain in the uncategorized section.

[Eärendil]

That's because the server caches all the uncategorized questions. If you categorize a question, then immediately come back to the uncategorized question page, chances are you'll still be looking at what's in the cache, as opposed to what's actually in the database. The cache should refresh every 5 minutes or so, though, so it'll eventually remove itself from the uncategorized page. This is a small nuisance I'd like to iron out but haven't quite figured out how to do it yet.