At The New Republic, intern Ezra Deutsch-Feldman makes a good point about the Man v. Computer battle on Jeopardy this week:
“Jeopardy!”is actually a terrible way of proving that Watson is more intelligent than its opponents. ... However, if the supercomputer triumphs, it will probably be for another reason entirely: because it can activate the buzzer most quickly.This is how the “Jeopardy!” rules work: Whoever buzzes in first—using a clicking device usually compared to a large pen—gets the first chance at answering the question. The wrinkle, however, is that the contestants have to wait until Alex Trebek is completely finished reading the question before they are allowed to buzz in. Buzz too soon, and your buzzer is “locked out” for a quarter of a second, giving opponents the chance to jump in and answer before you.
And in the recent test match between Watson, Jennings, and “Jeopardy!”champion Brad Rutter, none of the 15 questions were answered incorrectly by any of the players. In each case, the person who buzzed in first won the points associated with that clue. For all we know, all three players knew the answers to all the questions. Watson won that round, and it could easily have been because Watson was faster to the buzzer.
Indeed, when I called Watson’s creators to ask how the supercomputer controls its buzzer, they admitted that Watson does have a strong built-in advantage. According to David Shepler, who is IBM’s Challenge Program Manager for the Watson project, “The buzzer is enabled when the clue is done being read, when Alex Trebek gets to that last syllable, and the guy off stage pushes a button. That’s when people can buzz in, and at the same time a signal is sent to Watson saying the same thing—telling Watson that it can buzz in if it so desires.” This is akin to playing against an opponent with near-perfect reflexes.
The main difference between Jeopardy!, which I was on in 1994, and the old College Bowl game show, which I competed in from 1978-1982, is that on College Bowl toss-up questions you could buzz in while the announcer (Art Fleming, the original Jeopardy! moderator, when we went to Nationals in 1980) was reading the question. This made College Bowl less of a game of luck and reflexes than Jeopardy! and more a game a cognitive speed (albeit of a peculiar kind: the ability to figure out what a question was from the first few words).
The best strategy on College Bowl toss-ups was not to wait until you knew the answer, nor even to wait until you knew what the question was, but to buzz just before the moderator got to the key words that would reveal what the question was going to be. The announcer's momentum would carry him through the next word or two. You would then have a few seconds to A) figure out what the whole question was and B) what the answer was. If you got if right, your team of four players then got a bonus question where you'd get 10 seconds to answer. If you got the toss-up question wrong, the other team got to listen to the whole thing and then had five seconds to buzz in.
To give you an example of how this worked, and why College Bowl went off the air while Jeopardy! is still on, here's a question I remember from a practice round against the U. of Chicago at the 1980 Nationals:
Another example of why high level College Bowl became completely baffling to a mass audience: In the Rice U. championship game in 1979, my team was down by about 100 points with a couple of minutes left, which was kind of like being down by 9 points in the basketball (before the 3-point shot was introduced). I got the final five toss-up questions and we won by 5 points. The last toss-up question went:
Announcer: "What kind of victory --"
BUZZ
Guy on U. of Chicago team: "A Pyrrhic victory."
Art Fleming (unflappable as always, but still): "... Correct!"
Me to teammates [astonished]: "How did he get that?"
Teammate 1 [frustrated]: "What's wrong with this buzzer? I totally beat him to that!"
Me: "Huh? How?"
Teammate 2 [dismissively to me, while shaking his own balky buzzer]: "What other kind of victories are there?"
Another example of why high level College Bowl became completely baffling to a mass audience: In the Rice U. championship game in 1979, my team was down by about 100 points with a couple of minutes left, which was kind of like being down by 9 points in the basketball (before the 3-point shot was introduced). I got the final five toss-up questions and we won by 5 points. The last toss-up question went:
Announcer: "In the South Pacific, there's Bora-Bora; in the state of Wash--"
BUZZ
Me: [pausing for a couple of very long seconds while I tried to figure out why I buzzed in] ... Walla Walla.
In contrast, in Jeopardy!, you aren't allowed to buzz in early, so at the Ken Jennings-level of play, it's just a test of who buzzes in first after the arbitrary waiting period. This makes it more random, but a lot less baffling to viewers than College Bowl was.
The other notable thing about the College Bowl toss-up question was that it gave a huge advantage to youth. As a warm-up exhibition game before the 1980 Nationals, I recruited a well-balanced team of four Rice professors to play our team. One professor was Martin Wiener, now head of the history department at Rice and author of the 1981 book English Culture and the Decline of the Industrial Spirit: 1850-1980, which was to have a big influence on the Thatcher Government. Besides knowing vastly more than any of us callow undergrads, he had been on the 1966 College Bowl national championship team.
But, we undergrads dominated the toss-up questions and won in a rout.
It's pretty scary how much faster your brain was at 21.
Competing in high-school trivia, there was also a strong need for common sense and logic to answer questions.
ReplyDeleteThe most obvious questions if asked in a different manner, are often unmet except for by teammates of mine who were mathematically-apt.
Sheer knowledge however, is still king.
Trivia is by and large, a great way to meet intelligent, albeit pretentious people.
"a game a cognitive speed (albeit of a peculiar kind: the ability to figure out what a question was from the first few words). "
ReplyDeleteIt's basically Name That Tune for nerds.
The high school quiz bowl show It's Academic is still on the air in the DC metro area, with the original host from its start nearly 50 years ago.
At least in the late '90s it was still popular enough that you could get the cheerleaders to come out and support the buzzer jockeys. Quiz nerds get to hang out with cute bouncy girls, and they get to be on TV and show up their uglier and less athletic female rivals. Everybody was happy.
Steve, I checked out their archives and found your appearance on Jeopardy- you did well!
ReplyDeletehttp://m.youtube.com/#/watch?desktop_uri=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DoHg5SJYRHA0&v=oHg5SJYRHA0&gl=US&bmb=1
Dan in DC
I'm glad you posted this, Steve. I watched Jeopordy tonight and could almost see the frustration on the faces of the human competitors when they buzzed in a fraction of a second too late.
ReplyDeleteNot that I'm knocking Watson - the computer is a very good "answer engine" - but its main advantage over humans seems to be its reflexes.
The buzzing theory is flawed.
ReplyDeleteYou are assuming that Watson will know the answer as consistently as top players like Rutter and Jennings, who respond correctly 92 - 95% of the time. Watson just isn't that accurate as you can see from just this first match.
You are also wrong that humans are no match for a machine's "reflexes". Watch this sample where Watson is highly confident and quick in the answers, and is beaten to the buzzer time after time by another competitor (start at 1:20)
http://www.youtube.com/watch?v=7kOEmupSHB8
Trivia is by and large, a great way to meet intelligent, albeit pretentious people.
ReplyDeleteBut doesn't the huge nerd factor cancel out the positive trait of intelligence and intellectual curiosity?
You've just described why I never even tried to go on Jeopardy. I knew I could never master the timing of the buzzer, and could tell by watching that many contestants were losing purely by buzzer strategy.
ReplyDeleteThat, coupled with the fact that I'm nearsighted and sight plays a huge part in hearing, believe it or not (well, anyone who is nearsighted and doesn't wear their glasses will believe it) was enough to keep me from trying the show. Too bad, I'd otherwise have had a blast, win or lose.
It's Art Fleming (http://en.wikipedia.org/wiki/Art_Fleming). While I never competed on Jeopardy I watched it every day I during summer vacation plus whenever I was home sick.
ReplyDeleteWatson cleaned up on the easy questions, but the boys came back nicely as the questions got harder. So they should likely do better in Double Jeopardy. Watson appears to be just running searches on the text strings - the Beatles songs were right up Watson's alley, while he got skunked on the Decades category - probably had a hard time zeroing in on what exactly was being asked.
ReplyDeleteI believe Jeopardy! had different buzzing rules in the 60's. My mom was on the show then and I believe you could buzz as soon as the slide was lifted, although Art would always read the entire 'answer' before he went to the contestant for a response.
By the way - it's Art Fleming - which I'm sure you know, but you wrote Fletcher.
Steve, where you on the Rice team at the Southeast Regionals in 1979- held at the CDC in Atlanta? And did one of your teammates look like the guy who used to do the old Frito-Lay commercials?
ReplyDeleteP.S.- it's Art Fleming, not Fletcher.
"Art Fletcher, the original Jeopardy! moderator..."
ReplyDeleteYou meant Art Fleming, of course.
1979 Southeast Regionals -- at Tulane, in New Orleans?
ReplyDeleteWe got beat. Long drive home.
We had two 16-year-old freshmen on the 1979 team. One's now a math professor at William and Mary, the other is a professor at Columbia Law School. Those two guys were the core of the Rice team that went to the National final game in 1982 after I'd graduated.
ReplyDeleteJeopardy! is good TV but bad for deciding which contestant knows more trivia. Aside from the buzzer issue, there is too much randomness through the small sample size of daily doubles and the last question. Since the game usually depends on Final Jeopardy!, a strong player doesn't have overwhelming odds against a weak player.
ReplyDeleteThe best way to compare opponents is to have them each answer a large sample of identical questions in isolation, which would be horrific television. It was done in the last round of Win Ben Stein's money, but that show was carried by Jimmy Kimmel's one liners.
The best show I've seen for combining watchable TV with a fair test of ability was a special tournament for trivia champions hosted by Dennis Miller on the Game Show Network, called Grand Slam. Incidentally, Ken Jennings was the winner of this buzzerless competition.
http://www.youtube.com/watch?v=bq43_FYNzg0
Is trivia one of the last bastions of popular intellectual pursuits for Caucasians unchallenged by either:
ReplyDeletea) Indians who do very well in spelling bees and scrabble championships, or
b) NE Asians who dominate math competitions along with Jews (even the US team was 2/3 Asian) and classical music
If so, why? I can understand why the foreign language-speaking and more mathematically inclined NE Asians might not be as interested in trivia, but why aren't the purportedly more verbal native English-speaking Indians into it?
Isn't memorizing an encyclopedia for trivia similar to but more interesting than memorizing a dictionary for spelling bees? Are there too many pop cultural references in the US? Are trivia game shows popular in India or NE Asia?
Also, would non-socially engineered trivia contestants reflect the Wikipedia male/female split of 85/15 or more? I've only met a few aspy females interested in trivia minutia compared to men.
I've never heard of trivia becoming solely Caucasian from the two leagues I played.
ReplyDeleteOne notable school was stacked with wealthy Indian, Anglo and Italian private schoolers.
The rest of the player base were filled with a mishmash of Anglos, East European immigrants and Asians from public school.
"But doesn't the huge nerd factor cancel out the positive trait of intelligence and intellectual curiosity?"
ReplyDeleteOvercoming the nerdiness of every player is pretty easy once you engage in some friendly conversation. Nerds competing in those clubs actually made up the most interesting people I've met.
Although in all fairness, until the last two years of highschool, the people I tend to meet were typical party kids, club-goers, hip hop aficionados, etc.
'Teammate 2 [dismissively to me, while shaking his own balky buzzer]: "What other kind of victories are there?"'
ReplyDeleteFor some reason this is really funny to me. Good attitude to have in life.
That's funny. I've told that story a million times but I never noticed until you pointed it out that it works as a philosophy of life.
ReplyDeleteThat youtube link posted at the top of the thread - its not working for me.
ReplyDeleteThe rule should be that all buzzes within the first second (or whatever) are considered to be at the same time and any ties should be broken some other way.
ReplyDeleteYes, that makes sense, have the computer on Jeopardy randomly select from everybody who buzzes within one, maybe two seconds after Alex Trebek finishes the question.
ReplyDeleteMy favorite part of Jeopardy is when Alex Trabek has to pronounce a foreign phrase. All foreign phrases are pronounced in the Trabek dialect, regardless of the original language.
ReplyDeleteHow about computer vs computer?
ReplyDeleteAnon wrote:
ReplyDelete-----
The best way to compare opponents is to have them each answer a large sample of identical questions in isolation, which would be horrific television. It was done in the last round of Win Ben Stein's money, but that show was carried by Jimmy Kimmel's one liners.
--------
Isolation boxes is, or at least when I looked at TV, the standard format in my European country. Over here, we are big on fairness, and the absolutely dumbest part of USA TV do not get aired, or only aired on the low-brow non-state channels.
"Watch this sample where Watson is highly confident and quick in the answers, and is beaten to the buzzer time after time "
ReplyDeleteThat's completely false. Watch the video at 1:43 and go frame by frame. The contestant buzzed in while Watson was only at 23% confidence for the correct answer. A split second later, Watson reaches 97% confidence but by then it's too late.
You will see the exact same thing if you go frame by frame at 1:28. Watson had not yet reached confidence in the correct answer when the human buzzed in. Ditto for 1:36.
___________
Anyway Steve, I agree with you 100% percent and I made the same point with my wife last night while we were watching.
Watson's main advantage against these humans seems to be that a computer can move an actuator much faster than a human can push a button.
Watson is impressive but computers have a long way to go.
"Quiz nerds get to hang out with cute bouncy girls, and they get to be on TV and show up their uglier and less athletic female rivals. Everybody was happy."
ReplyDeleteMaybe this is an east coast-heartland thing (and I'd love to know if nerds in different areas have different aspirations), but how about the nerdier girls we might actually be able to keep? Or do these only exist in college towns?
"while he got skunked on the Decades category - probably had a hard time zeroing in on what exactly was being asked."
ReplyDeleteI agree, and this was rather disappointing. As I recall, the category was called something like "name the decade." So it's very basic, simple human reasoning to figure out what kind of answer is being sought by each question. Also, human flexibility allows us to come up with a simple algorythm on the fly for answering this type of question and then applying it to the remaining questions in the category.
My conclusion is that computers have a long way to go before they are competitive with humans in terms of reasoning ability and flexibility.
Years ago in a newspaper I read a sad tale about a fellow in Sydney who'd been banned from all Trivial Pursuit contests in that city because, having devoted his entire adult non-life to reading reference books, he was better by light years than any other contestant. Of course he had the usual scientific and mass-media-related questions down pat, but apparently you could also ask him some totally gonzo question like "Who was President of Honduras in 1931?" and he'd know the correct answer. I sometimes wonder how he would've coped in a TV quiz environment.
ReplyDeleteAccording to another interview with a Watson creator, getting Watson to buzz in correctly was actually half of the problem- it would be able to answer most of the questions within 3 seconds of the end of the question, but the humans were able to know whether or not they could answer the question really early on, and so would beat Watson to the buzz almost every time.
ReplyDeleteThey have mostly solved that problem now.
The Jeopardy category I did best on was "1911."
ReplyDeleteI've always found that knowing the exact date of things is very helpful in doing reality checks on arguments about causality. If somebody says "X caused Y" but X happened after Y, out it goes.
A big reason Paul Johnson's 1983 book "Modern Times" had such an impact on readers was its obsession with getting the exact dates down in print. In a world connected by telegraph and telephone, exact dates are extremely useful in constructing cause-and-effect narratives.
I have a lot of pet peeves: here's another one:
ReplyDelete"... between Watson, Jennings, and “Jeopardy!”champion Brad Rutter ..."
Has "among" disappeared from the English language? The derivation of "between" obviously impies the existence of two, and only two, entities for somebody or something to be between. Watson was between Jenning and Rudder, but the contest was not between the three of them. And don't tell me that the dictionary says otherwise; that's another pet peeve.
OK. OK. I'll stop now.
"...between Watson, Jennings, and “Jeopardy!”champion Brad Rutter..." I think I'm the only person left who uses the word "among." I'm also the only person left who trys todo something. Everyone else tells me they'll try anddo it.
ReplyDeleteRe: kinds of victory. Is anything abject other than poverty?
Jim O: failure.
ReplyDeleteDan in DC: How's about checking that URL?
"TV Presenters for $1,000"
ReplyDelete"Art Fletcher..."
Bzzzz
"Who was not a former presenter on 'Jeopardy'?"
"Jim O said...
ReplyDeleteRe: kinds of victory. Is anything abject other than poverty?"
Failure. Misery.
Does this mean that IBM will now replace Google?
ReplyDeleteAre you the same robert61 who once wrote "be advised that 'ungrammatical' is not just a classy synonym for 'f—in stupid.'"
ReplyDeleteI've never heard of trivia becoming solely Caucasian from the two leagues I played.
ReplyDeleteNot solely, but with a higher % of Caucasians relative to somewhat similar competitions like spelling, scrabble, classical music and math.
I have met a lot more white guys who love trivia much more than things like spelling or classical music.
How do these quiz shows avoid the pitfall of falling below EEOC's 4/5 proportional representation of all ethnics?
Haven't they ever been sued giving all those cash prizes to such narrow demographics?
" ...but how about the nerdier girls we might actually be able to keep? Or do these only exist in college towns?"
ReplyDeleteNerdy girls do not exist. Some will call themselves nerdy if they sense that it's what you want to hear from them, but that doesn't make them nerdy.
"Is anything abject other than poverty?"
ReplyDeleteFailure.
I watched the College bowl on TV about twice. The first time I competed with all the teams and I usually got more answers than any of the teams. The second time I fully expected to beat every team every time but I became upset whenever I missed a question or I wasn't fast enough.
ReplyDeleteI never watched a third time. It was too unsettling. I remembered that experience and have never watched Jeopardy or any other quiz show since. I can't handle the pressure.
It's called the Boob Tube. Working your brain extra hard on TV just seems wrong.
Albertosaurus
Abject deference, which a lot of people are demanding these days.
ReplyDeleteThat's basically what Ken Jennings writes in his FAQ:
ReplyDelete"Most of the contestants can answer most of the questions. But Jeopardy! victory goes not to the biggest brain—it goes to the smoothest thumb. Timing on the tricky Jeopardy! buzzer is often what separates the winner from the, well, non-winners, and the Jeopardy! buzzer is a cruel mistress." (http://www.ken-jennings.com/faqjeopardy.html)
Instead of the usual canned and uninformative Presidential debates, perhaps there should be a Jeopardy, Decision 2012 Edition, where candidates demonstrate knowledge of US History, Government, and Economics. There would be primary rounds, followed by the Presidential round.
ReplyDeleteOr how about, Jeopardy, Supreme Court Edition. Sonya Sotomayor standing at the podium with $0 at the end of the game might actually have persuaded some Democrat Senators to vote against her.
Indians who do very well in spelling bees and scrabble championships...Isn't memorizing an encyclopedia for trivia similar to but more interesting than memorizing a dictionary for spelling bees?
ReplyDeleteTrivia is something smart people pick up in the process of living life. General knowledge and g have a decently high correlation. If Indians were particularly bright, they would learn how to spell without spending hours memorizing spellings, and at the same time would pick up lots of facts.
"Nerdy girls do not exist. Some will call themselves nerdy if they sense that it's what you want to hear from them, but that doesn't make them nerdy."
ReplyDeleteDepends on what your qualifications for "nerdiness" entail. I would agree that there are probably no more than a handful of female science nerds. But if you're talking about something along the lines of medieval english history, then you'll find a significant number of female "nerds."
@rob and @hbd trivia.
ReplyDeleteIndians in Jeopardy
http://en.wikipedia.org/wiki/Jeopardy!_Tournament_of_Champions#List_of_participants
season 26 winner, May 2010 = Vijay Balse, semi-finalist = Andy Srinivasan.
--
http://en.wikipedia.org/wiki/Jeopardy!_College_Championship
season 18, Nov-2001 winner = Vinita Kailasanath
season 26, Feb-2010, 2nd runner up = Surya Sabapathy
season 27, nov-2010 , semi-finalist = Sid Chandrasekhar
--
http://en.wikipedia.org/wiki/Jeopardy!_Teen_Tournament
season-22, feb-2006, winner = Papa Chakravorthy
season-24, feb-2008, semi-finalist = Naren Tallapragda
season-25, nov-2008, winner = Anurag Kashyap,
semi-finalist = karan thakar.
season-26, nov-2009, semifinalist = Hema Karunakaram
"Also, would non-socially engineered trivia contestants reflect the Wikipedia male/female split of 85/15 or more? I've only met a few aspy females interested in trivia minutia compared to men."
ReplyDeleteI was on a quiz bowl team in high school and participated in county-wide "meets" every couple of weeks. The female-male split was 40-60, although admittedly amongst the five people ("starters") who answered most of the questions for each team it closer to 25-75.
Anyway, I've always enjoyed trivia. I was able to beat people who were likely smarter than me =). I would love to give jeopardy a shot one day.
"It's pretty scary how much faster your brain was at 21."
ReplyDeleteYeah but some other things that they do other things faster they probably don't want to...
Steve, I remember going up against Rice in 79, and I'm sure it was at the Center for Disease Control building (no idea why) in Atlanta. I remember one guy on the Rice team who looked like Gene Shalit, whom I always confused with the guy who did the Frito-Lay commercials.....
ReplyDeleteI also remember a meet in 1980; I'm thinking Atlanta,and I vividly remember Lloyd Bush from the Tulane team running in after a match to announce that the U.S hockey team had beaten the Soviets. Do you remember him? I'm pretty sure it wasn't at Tulane, because one of my teammates had done his undergrad there, and I would have remembered him showing us the town.
I also remember a College Bowl radio show with Art Fleming moderating, that was at Farleigh Dickinson in New Jersey the following fall (1980). I believe this was an invitation-only meet of some type- the New York Times sent a reporter. I don't remember Rice being there, but it's been a few years since then!
It sounds like we overlapped, but I'm not sure where. I don't remember going to Atlanta for College Bowl. I remember I was at the College Bowl regional tournament in North Texas (TCU?) in 1980 when we heard the Americans had beaten the Soviets in Olympic hockey.
ReplyDeleteWe went to the national tournament in 1980 at Marshall in Huntington, West Virginia but got beat pretty fast.
I went to the invitational tournament in New Jersey in the fall of 1979 but had graduated from Rice by the fall of 1980. Fairleigh Dickinson was the college?
It's funny how vague we are about memories of our own lives. After every College Bowl game you had to give you Social Security Number. I always pulled out my SSN card because I hadn't memorized it.
I'm guessing I looked like Gene Shalit in 1979-1980: glasses, mustache, and curly hair.
(Laughs) Yep, it is strange how we remember things. Ah well, it was fun. I guess that might have been you with the glasses and mustache. Were they black thick framed glasses? Anyway, Rice beat us. You're right about buzzer strategies, my team had worked out a strategy that determined how aggressive we should be against other teams by how quick they answered- we figured that it was better to jump in early against some teams, and less so against others. We didn't put in the statistical analysis required to optimize such a strategy, as data collection was obviously a problem- unless you watched each match in real time, you wouldn't know which strategy to use, as that type of data was not reflected in the win/lose column distributed after the fact.
ReplyDeleteDid you ever go up against MIT? we kind of considered them egotistical pricks......although I guess we all were, somewhat!
Dan in DC: I went to the trouble of cleaning up your URL.
ReplyDeleteThanks, buddy. Should have known.
Even if Watson had some advantage in buzzing, anyway he also knew the right answers, soit is AT LEAST at same level as the world champion. Moreover also being faster is an example of superiority.
ReplyDelete