February 14, 2011

Jeopardy!

At The New Republic, intern Ezra Deutsch-Feldman makes a good point about the Man v. Computer battle on Jeopardy this week:

“Jeopardy!”is actually a terrible way of proving that Watson is more intelligent than its opponents. ... However, if the supercomputer triumphs, it will probably be for another reason entirely: because it can activate the buzzer most quickly.

This is how the “Jeopardy!” rules work: Whoever buzzes in first—using a clicking device usually compared to a large pen—gets the first chance at answering the question. The wrinkle, however, is that the contestants have to wait until Alex Trebek is completely finished reading the question before they are allowed to buzz in. Buzz too soon, and your buzzer is “locked out” for a quarter of a second, giving opponents the chance to jump in and answer before you.

And in the recent test match between Watson, Jennings, and “Jeopardy!”champion Brad Rutter, none of the 15 questions were answered incorrectly by any of the players. In each case, the person who buzzed in first won the points associated with that clue. For all we know, all three players knew the answers to all the questions. Watson won that round, and it could easily have been because Watson was faster to the buzzer.

Indeed, when I called Watson’s creators to ask how the supercomputer controls its buzzer, they admitted that Watson does have a strong built-in advantage. According to David Shepler, who is IBM’s Challenge Program Manager for the Watson project, “The buzzer is enabled when the clue is done being read, when Alex Trebek gets to that last syllable, and the guy off stage pushes a button. That’s when people can buzz in, and at the same time a signal is sent to Watson saying the same thing—telling Watson that it can buzz in if it so desires.” This is akin to playing against an opponent with near-perfect reflexes.

The main difference between Jeopardy!, which I was on in 1994, and the old College Bowl game show, which I competed in from 1978-1982, is that on College Bowl toss-up questions you could buzz in while the announcer (Art Fleming, the original Jeopardy! moderator, when we went to Nationals in 1980) was reading the question. This made College Bowl less of a game of luck and reflexes than Jeopardy! and more a game a cognitive speed (albeit of a peculiar kind: the ability to figure out what a question was from the first few words). 

The best strategy on College Bowl toss-ups was not to wait until you knew the answer, nor even to wait until you knew what the question was, but to buzz just before the moderator got to the key words that would reveal what the question was going to be. The announcer's momentum would carry him through the next word or two. You would then have a few seconds to A) figure out what the whole question was and B) what the answer was. If you got if right, your team of four players then got a bonus question where you'd get 10 seconds to answer. If you got the toss-up question wrong, the other team got to listen to the whole thing and then had five seconds to buzz in. 

To give you an example of how this worked, and why College Bowl went off the air while Jeopardy! is still on, here's a question I remember from a practice round against the U. of Chicago at the 1980 Nationals:
Announcer: "What kind of victory --"

BUZZ

Guy on U. of Chicago team: "A Pyrrhic victory."

Art Fleming (unflappable as always, but still): "... Correct!"
Me to teammates [astonished]: "How did he get that?"

Teammate 1 [frustrated]: "What's wrong with this buzzer? I totally beat him to that!"
Me: "Huh? How?"

Teammate 2 [dismissively to me, while shaking his own balky buzzer]: "What other kind of victories are there?"

Another example of why high level College Bowl became completely baffling to a mass audience: In the Rice U. championship game in 1979, my team was down by about 100 points with a couple of minutes left, which was kind of like being down by 9 points in the basketball (before the 3-point shot was introduced). I got the final five toss-up questions and we won by 5 points. The last toss-up question went:
Announcer: "In the South Pacific, there's Bora-Bora; in the state of Wash--"

BUZZ

Me: [pausing for a couple of very long seconds while I tried to figure out why I buzzed in] ... Walla Walla.

In contrast, in Jeopardy!, you aren't allowed to buzz in early, so at the Ken Jennings-level of play, it's just a test of who buzzes in first after the arbitrary waiting period. This makes it more random, but a lot less baffling to viewers than College Bowl was. 

The other notable thing about the College Bowl toss-up question was that it gave a huge advantage to youth. As a warm-up exhibition game before the 1980 Nationals, I recruited a well-balanced team of four Rice professors to play our team. One professor was Martin Wiener, now head of the history department at Rice and author of the 1981 book English Culture and the Decline of the Industrial Spirit: 1850-1980, which was to have a big influence on the Thatcher Government. Besides knowing vastly more than any of us callow undergrads, he had been on the 1966 College Bowl national championship team. 

But, we undergrads dominated the toss-up questions and won in a rout. 

It's pretty scary how much faster your brain was at 21.

55 comments:

  1. Competing in high-school trivia, there was also a strong need for common sense and logic to answer questions.

    The most obvious questions if asked in a different manner, are often unmet except for by teammates of mine who were mathematically-apt.

    Sheer knowledge however, is still king.

    Trivia is by and large, a great way to meet intelligent, albeit pretentious people.

    ReplyDelete
  2. "a game a cognitive speed (albeit of a peculiar kind: the ability to figure out what a question was from the first few words). "

    It's basically Name That Tune for nerds.

    The high school quiz bowl show It's Academic is still on the air in the DC metro area, with the original host from its start nearly 50 years ago.

    At least in the late '90s it was still popular enough that you could get the cheerleaders to come out and support the buzzer jockeys. Quiz nerds get to hang out with cute bouncy girls, and they get to be on TV and show up their uglier and less athletic female rivals. Everybody was happy.

    ReplyDelete
  3. Steve, I checked out their archives and found your appearance on Jeopardy- you did well!

    http://m.youtube.com/#/watch?desktop_uri=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DoHg5SJYRHA0&v=oHg5SJYRHA0&gl=US&bmb=1

    Dan in DC

    ReplyDelete
  4. I'm glad you posted this, Steve. I watched Jeopordy tonight and could almost see the frustration on the faces of the human competitors when they buzzed in a fraction of a second too late.

    Not that I'm knocking Watson - the computer is a very good "answer engine" - but its main advantage over humans seems to be its reflexes.

    ReplyDelete
  5. The buzzing theory is flawed.

    You are assuming that Watson will know the answer as consistently as top players like Rutter and Jennings, who respond correctly 92 - 95% of the time. Watson just isn't that accurate as you can see from just this first match.

    You are also wrong that humans are no match for a machine's "reflexes". Watch this sample where Watson is highly confident and quick in the answers, and is beaten to the buzzer time after time by another competitor (start at 1:20)

    http://www.youtube.com/watch?v=7kOEmupSHB8

    ReplyDelete
  6. Trivia is by and large, a great way to meet intelligent, albeit pretentious people.

    But doesn't the huge nerd factor cancel out the positive trait of intelligence and intellectual curiosity?

    ReplyDelete
  7. You've just described why I never even tried to go on Jeopardy. I knew I could never master the timing of the buzzer, and could tell by watching that many contestants were losing purely by buzzer strategy.

    That, coupled with the fact that I'm nearsighted and sight plays a huge part in hearing, believe it or not (well, anyone who is nearsighted and doesn't wear their glasses will believe it) was enough to keep me from trying the show. Too bad, I'd otherwise have had a blast, win or lose.

    ReplyDelete
  8. It's Art Fleming (http://en.wikipedia.org/wiki/Art_Fleming). While I never competed on Jeopardy I watched it every day I during summer vacation plus whenever I was home sick.

    ReplyDelete
  9. Watson cleaned up on the easy questions, but the boys came back nicely as the questions got harder. So they should likely do better in Double Jeopardy. Watson appears to be just running searches on the text strings - the Beatles songs were right up Watson's alley, while he got skunked on the Decades category - probably had a hard time zeroing in on what exactly was being asked.

    I believe Jeopardy! had different buzzing rules in the 60's. My mom was on the show then and I believe you could buzz as soon as the slide was lifted, although Art would always read the entire 'answer' before he went to the contestant for a response.


    By the way - it's Art Fleming - which I'm sure you know, but you wrote Fletcher.

    ReplyDelete
  10. Steve, where you on the Rice team at the Southeast Regionals in 1979- held at the CDC in Atlanta? And did one of your teammates look like the guy who used to do the old Frito-Lay commercials?

    P.S.- it's Art Fleming, not Fletcher.

    ReplyDelete
  11. "Art Fletcher, the original Jeopardy! moderator..."

    You meant Art Fleming, of course.

    ReplyDelete
  12. 1979 Southeast Regionals -- at Tulane, in New Orleans?

    We got beat. Long drive home.

    ReplyDelete
  13. We had two 16-year-old freshmen on the 1979 team. One's now a math professor at William and Mary, the other is a professor at Columbia Law School. Those two guys were the core of the Rice team that went to the National final game in 1982 after I'd graduated.

    ReplyDelete
  14. Jeopardy! is good TV but bad for deciding which contestant knows more trivia. Aside from the buzzer issue, there is too much randomness through the small sample size of daily doubles and the last question. Since the game usually depends on Final Jeopardy!, a strong player doesn't have overwhelming odds against a weak player.

    The best way to compare opponents is to have them each answer a large sample of identical questions in isolation, which would be horrific television. It was done in the last round of Win Ben Stein's money, but that show was carried by Jimmy Kimmel's one liners.

    The best show I've seen for combining watchable TV with a fair test of ability was a special tournament for trivia champions hosted by Dennis Miller on the Game Show Network, called Grand Slam. Incidentally, Ken Jennings was the winner of this buzzerless competition.

    http://www.youtube.com/watch?v=bq43_FYNzg0

    ReplyDelete
  15. HBD and Trivia for the block Wink2/14/11, 9:25 PM

    Is trivia one of the last bastions of popular intellectual pursuits for Caucasians unchallenged by either:

    a) Indians who do very well in spelling bees and scrabble championships, or

    b) NE Asians who dominate math competitions along with Jews (even the US team was 2/3 Asian) and classical music

    If so, why? I can understand why the foreign language-speaking and more mathematically inclined NE Asians might not be as interested in trivia, but why aren't the purportedly more verbal native English-speaking Indians into it?

    Isn't memorizing an encyclopedia for trivia similar to but more interesting than memorizing a dictionary for spelling bees? Are there too many pop cultural references in the US? Are trivia game shows popular in India or NE Asia?

    Also, would non-socially engineered trivia contestants reflect the Wikipedia male/female split of 85/15 or more? I've only met a few aspy females interested in trivia minutia compared to men.

    ReplyDelete
  16. I've never heard of trivia becoming solely Caucasian from the two leagues I played.

    One notable school was stacked with wealthy Indian, Anglo and Italian private schoolers.

    The rest of the player base were filled with a mishmash of Anglos, East European immigrants and Asians from public school.

    ReplyDelete
  17. "But doesn't the huge nerd factor cancel out the positive trait of intelligence and intellectual curiosity?"

    Overcoming the nerdiness of every player is pretty easy once you engage in some friendly conversation. Nerds competing in those clubs actually made up the most interesting people I've met.

    Although in all fairness, until the last two years of highschool, the people I tend to meet were typical party kids, club-goers, hip hop aficionados, etc.

    ReplyDelete
  18. 'Teammate 2 [dismissively to me, while shaking his own balky buzzer]: "What other kind of victories are there?"'

    For some reason this is really funny to me. Good attitude to have in life.

    ReplyDelete
  19. That's funny. I've told that story a million times but I never noticed until you pointed it out that it works as a philosophy of life.

    ReplyDelete
  20. That youtube link posted at the top of the thread - its not working for me.

    ReplyDelete
  21. The rule should be that all buzzes within the first second (or whatever) are considered to be at the same time and any ties should be broken some other way.

    ReplyDelete
  22. Yes, that makes sense, have the computer on Jeopardy randomly select from everybody who buzzes within one, maybe two seconds after Alex Trebek finishes the question.

    ReplyDelete
  23. My favorite part of Jeopardy is when Alex Trabek has to pronounce a foreign phrase. All foreign phrases are pronounced in the Trabek dialect, regardless of the original language.

    ReplyDelete
  24. How about computer vs computer?

    ReplyDelete
  25. Anon wrote:
    -----
    The best way to compare opponents is to have them each answer a large sample of identical questions in isolation, which would be horrific television. It was done in the last round of Win Ben Stein's money, but that show was carried by Jimmy Kimmel's one liners.
    --------
    Isolation boxes is, or at least when I looked at TV, the standard format in my European country. Over here, we are big on fairness, and the absolutely dumbest part of USA TV do not get aired, or only aired on the low-brow non-state channels.

    ReplyDelete
  26. "Watch this sample where Watson is highly confident and quick in the answers, and is beaten to the buzzer time after time "

    That's completely false. Watch the video at 1:43 and go frame by frame. The contestant buzzed in while Watson was only at 23% confidence for the correct answer. A split second later, Watson reaches 97% confidence but by then it's too late.

    You will see the exact same thing if you go frame by frame at 1:28. Watson had not yet reached confidence in the correct answer when the human buzzed in. Ditto for 1:36.

    ___________

    Anyway Steve, I agree with you 100% percent and I made the same point with my wife last night while we were watching.

    Watson's main advantage against these humans seems to be that a computer can move an actuator much faster than a human can push a button.

    Watson is impressive but computers have a long way to go.

    ReplyDelete
  27. "Quiz nerds get to hang out with cute bouncy girls, and they get to be on TV and show up their uglier and less athletic female rivals. Everybody was happy."

    Maybe this is an east coast-heartland thing (and I'd love to know if nerds in different areas have different aspirations), but how about the nerdier girls we might actually be able to keep? Or do these only exist in college towns?

    ReplyDelete
  28. "while he got skunked on the Decades category - probably had a hard time zeroing in on what exactly was being asked."

    I agree, and this was rather disappointing. As I recall, the category was called something like "name the decade." So it's very basic, simple human reasoning to figure out what kind of answer is being sought by each question. Also, human flexibility allows us to come up with a simple algorythm on the fly for answering this type of question and then applying it to the remaining questions in the category.

    My conclusion is that computers have a long way to go before they are competitive with humans in terms of reasoning ability and flexibility.

    ReplyDelete
  29. Years ago in a newspaper I read a sad tale about a fellow in Sydney who'd been banned from all Trivial Pursuit contests in that city because, having devoted his entire adult non-life to reading reference books, he was better by light years than any other contestant. Of course he had the usual scientific and mass-media-related questions down pat, but apparently you could also ask him some totally gonzo question like "Who was President of Honduras in 1931?" and he'd know the correct answer. I sometimes wonder how he would've coped in a TV quiz environment.

    ReplyDelete
  30. According to another interview with a Watson creator, getting Watson to buzz in correctly was actually half of the problem- it would be able to answer most of the questions within 3 seconds of the end of the question, but the humans were able to know whether or not they could answer the question really early on, and so would beat Watson to the buzz almost every time.

    They have mostly solved that problem now.

    ReplyDelete
  31. The Jeopardy category I did best on was "1911."

    I've always found that knowing the exact date of things is very helpful in doing reality checks on arguments about causality. If somebody says "X caused Y" but X happened after Y, out it goes.

    A big reason Paul Johnson's 1983 book "Modern Times" had such an impact on readers was its obsession with getting the exact dates down in print. In a world connected by telegraph and telephone, exact dates are extremely useful in constructing cause-and-effect narratives.

    ReplyDelete
  32. I have a lot of pet peeves: here's another one:

    "... between Watson, Jennings, and “Jeopardy!”champion Brad Rutter ..."

    Has "among" disappeared from the English language? The derivation of "between" obviously impies the existence of two, and only two, entities for somebody or something to be between. Watson was between Jenning and Rudder, but the contest was not between the three of them. And don't tell me that the dictionary says otherwise; that's another pet peeve.

    OK. OK. I'll stop now.

    ReplyDelete
  33. "...between Watson, Jennings, and “Jeopardy!”champion Brad Rutter..." I think I'm the only person left who uses the word "among." I'm also the only person left who trys todo something. Everyone else tells me they'll try anddo it.

    Re: kinds of victory. Is anything abject other than poverty?

    ReplyDelete
  34. Jim O: failure.

    Dan in DC: How's about checking that URL?

    ReplyDelete
  35. "TV Presenters for $1,000"

    "Art Fletcher..."

    Bzzzz

    "Who was not a former presenter on 'Jeopardy'?"

    ReplyDelete
  36. "Jim O said...

    Re: kinds of victory. Is anything abject other than poverty?"

    Failure. Misery.

    ReplyDelete
  37. Does this mean that IBM will now replace Google?

    ReplyDelete
  38. Are you the same robert61 who once wrote "be advised that 'ungrammatical' is not just a classy synonym for 'f—in stupid.'"

    ReplyDelete
  39. HBD and trivia for the block Wink2/15/11, 8:52 AM

    I've never heard of trivia becoming solely Caucasian from the two leagues I played.

    Not solely, but with a higher % of Caucasians relative to somewhat similar competitions like spelling, scrabble, classical music and math.

    I have met a lot more white guys who love trivia much more than things like spelling or classical music.

    How do these quiz shows avoid the pitfall of falling below EEOC's 4/5 proportional representation of all ethnics?

    Haven't they ever been sued giving all those cash prizes to such narrow demographics?

    ReplyDelete
  40. " ...but how about the nerdier girls we might actually be able to keep? Or do these only exist in college towns?"

    Nerdy girls do not exist. Some will call themselves nerdy if they sense that it's what you want to hear from them, but that doesn't make them nerdy.

    ReplyDelete
  41. "Is anything abject other than poverty?"

    Failure.

    ReplyDelete
  42. I watched the College bowl on TV about twice. The first time I competed with all the teams and I usually got more answers than any of the teams. The second time I fully expected to beat every team every time but I became upset whenever I missed a question or I wasn't fast enough.

    I never watched a third time. It was too unsettling. I remembered that experience and have never watched Jeopardy or any other quiz show since. I can't handle the pressure.

    It's called the Boob Tube. Working your brain extra hard on TV just seems wrong.

    Albertosaurus

    ReplyDelete
  43. helene edwards2/15/11, 10:52 AM

    Abject deference, which a lot of people are demanding these days.

    ReplyDelete
  44. That's basically what Ken Jennings writes in his FAQ:

    "Most of the contestants can answer most of the questions. But Jeopardy! victory goes not to the biggest brain—it goes to the smoothest thumb. Timing on the tricky Jeopardy! buzzer is often what separates the winner from the, well, non-winners, and the Jeopardy! buzzer is a cruel mistress." (http://www.ken-jennings.com/faqjeopardy.html)

    ReplyDelete
  45. Instead of the usual canned and uninformative Presidential debates, perhaps there should be a Jeopardy, Decision 2012 Edition, where candidates demonstrate knowledge of US History, Government, and Economics. There would be primary rounds, followed by the Presidential round.

    Or how about, Jeopardy, Supreme Court Edition. Sonya Sotomayor standing at the podium with $0 at the end of the game might actually have persuaded some Democrat Senators to vote against her.

    ReplyDelete
  46. Indians who do very well in spelling bees and scrabble championships...Isn't memorizing an encyclopedia for trivia similar to but more interesting than memorizing a dictionary for spelling bees?

    Trivia is something smart people pick up in the process of living life. General knowledge and g have a decently high correlation. If Indians were particularly bright, they would learn how to spell without spending hours memorizing spellings, and at the same time would pick up lots of facts.

    ReplyDelete
  47. "Nerdy girls do not exist. Some will call themselves nerdy if they sense that it's what you want to hear from them, but that doesn't make them nerdy."

    Depends on what your qualifications for "nerdiness" entail. I would agree that there are probably no more than a handful of female science nerds. But if you're talking about something along the lines of medieval english history, then you'll find a significant number of female "nerds."

    ReplyDelete
  48. @rob and @hbd trivia.

    Indians in Jeopardy

    http://en.wikipedia.org/wiki/Jeopardy!_Tournament_of_Champions#List_of_participants

    season 26 winner, May 2010 = Vijay Balse, semi-finalist = Andy Srinivasan.

    --

    http://en.wikipedia.org/wiki/Jeopardy!_College_Championship

    season 18, Nov-2001 winner = Vinita Kailasanath

    season 26, Feb-2010, 2nd runner up = Surya Sabapathy

    season 27, nov-2010 , semi-finalist = Sid Chandrasekhar

    --

    http://en.wikipedia.org/wiki/Jeopardy!_Teen_Tournament

    season-22, feb-2006, winner = Papa Chakravorthy

    season-24, feb-2008, semi-finalist = Naren Tallapragda

    season-25, nov-2008, winner = Anurag Kashyap,
    semi-finalist = karan thakar.

    season-26, nov-2009, semifinalist = Hema Karunakaram

    ReplyDelete
  49. "Also, would non-socially engineered trivia contestants reflect the Wikipedia male/female split of 85/15 or more? I've only met a few aspy females interested in trivia minutia compared to men."

    I was on a quiz bowl team in high school and participated in county-wide "meets" every couple of weeks. The female-male split was 40-60, although admittedly amongst the five people ("starters") who answered most of the questions for each team it closer to 25-75.

    Anyway, I've always enjoyed trivia. I was able to beat people who were likely smarter than me =). I would love to give jeopardy a shot one day.

    ReplyDelete
  50. "It's pretty scary how much faster your brain was at 21."

    Yeah but some other things that they do other things faster they probably don't want to...

    ReplyDelete
  51. Steve, I remember going up against Rice in 79, and I'm sure it was at the Center for Disease Control building (no idea why) in Atlanta. I remember one guy on the Rice team who looked like Gene Shalit, whom I always confused with the guy who did the Frito-Lay commercials.....

    I also remember a meet in 1980; I'm thinking Atlanta,and I vividly remember Lloyd Bush from the Tulane team running in after a match to announce that the U.S hockey team had beaten the Soviets. Do you remember him? I'm pretty sure it wasn't at Tulane, because one of my teammates had done his undergrad there, and I would have remembered him showing us the town.

    I also remember a College Bowl radio show with Art Fleming moderating, that was at Farleigh Dickinson in New Jersey the following fall (1980). I believe this was an invitation-only meet of some type- the New York Times sent a reporter. I don't remember Rice being there, but it's been a few years since then!

    ReplyDelete
  52. It sounds like we overlapped, but I'm not sure where. I don't remember going to Atlanta for College Bowl. I remember I was at the College Bowl regional tournament in North Texas (TCU?) in 1980 when we heard the Americans had beaten the Soviets in Olympic hockey.

    We went to the national tournament in 1980 at Marshall in Huntington, West Virginia but got beat pretty fast.

    I went to the invitational tournament in New Jersey in the fall of 1979 but had graduated from Rice by the fall of 1980. Fairleigh Dickinson was the college?

    It's funny how vague we are about memories of our own lives. After every College Bowl game you had to give you Social Security Number. I always pulled out my SSN card because I hadn't memorized it.

    I'm guessing I looked like Gene Shalit in 1979-1980: glasses, mustache, and curly hair.

    ReplyDelete
  53. (Laughs) Yep, it is strange how we remember things. Ah well, it was fun. I guess that might have been you with the glasses and mustache. Were they black thick framed glasses? Anyway, Rice beat us. You're right about buzzer strategies, my team had worked out a strategy that determined how aggressive we should be against other teams by how quick they answered- we figured that it was better to jump in early against some teams, and less so against others. We didn't put in the statistical analysis required to optimize such a strategy, as data collection was obviously a problem- unless you watched each match in real time, you wouldn't know which strategy to use, as that type of data was not reflected in the win/lose column distributed after the fact.

    Did you ever go up against MIT? we kind of considered them egotistical pricks......although I guess we all were, somewhat!

    ReplyDelete
  54. Dan in DC: I went to the trouble of cleaning up your URL.

    Thanks, buddy. Should have known.

    ReplyDelete
  55. Even if Watson had some advantage in buzzing, anyway he also knew the right answers, soit is AT LEAST at same level as the world champion. Moreover also being faster is an example of superiority.

    ReplyDelete

Comments are moderated, at whim.