'Jeopardy!' Man vs. Machine Match Draws Crowd at MIT

'Jeopardy!' Man vs. Machine Match Draws Crowd at MIT

By Karen Klinger

Answer: It's the size of 10 refrigerators, has a 15-trillion byte memory, is estimated to have cost up to $2 billion and when it plays "Jeopardy!" it is represented on stage by an avatar and uses a mechanical finger to press a buzzer.

Question: What is Watson, IBM's supercomputer, battling against the two best human "Jeopardy!" players ever in a much-hyped showdown of man versus machine? The contest drew a large crowd February 14 to an MIT auditorium at one of the universities that has played a key role in a development process that aims to demonstrate a computer's ability to use and understand the kind of "natural language" that is innate in humans, but difficult for even the most advanced electronic device to master.

At the end of the first round of a three-night, two-game tournament, Watson was tied with "Jeopardy!'s" all-time top money winner Brad Rutter (earnings of $3.25 million) at $5,000, with Ken Jennings, who won a record 74 games and $2.5 million in 2004-2005, bringing up the rear at $2,000. The games were prerecorded at IBM's research center in Yorktown Heights, N.Y., with the results kept secret until the match's conclusion February 16. The winner will take home $1 million, with the runners-up getting $300,000 for second place and $200,000 for third. I.B.M. said it will give all of Watson's winnings to charity, while Rutter and Jennings will donate half of what they win.

IBM Researcher David Gondek

David Gondek, an IBM research scientist and one of the team members who has spent four years preparing Watson (named for IBM's founder, Thomas J. Watson) for its big moment on national television, was at the MIT event to shed light on how the computer has been prepped to understand and answer "Jeopardy!" questions, and what the technology might be used for beyond game shows. Over the years, he said, researchers have loaded the computer with hundreds of millions of pages of every sort of information they thought could be useful, all the while watching a lot of "Jeopardy!" contests ("We had nothing else to do," Gondek joked).

But while Watson is very good, and very fast, at using algorithms to retrieve straightforward facts and figures to come up with answers (the same way Google uses an algorithmic system), Gondek said its weakness--like that of all computers--is matching the intuitive understanding of linguistic nuances that humans are good at. He said "Jeopardy!" posed a great challenge because of the way the show uses puns, rhymes, riddles and humor in its trademark question-and-answer format, which human players can easily grasp, but can be a struggle for non-human competitors.

To prepare for the man-machine showdown, the IBM team put Watson up against former "Jeopardy!" contestants in 55 games, which the computer won 71 percent of the time. But when the computer struggled, it was often because it could not make linguistic leaps of inference that would be obvious to a good human player. Among Gondek's examples: Under the category "New York Times Headlines," a clue said an exclamation point would be warranted "for the end! of this in 1918." The right answer was World War I. But Watson's response was: "a sentence."


Another question asked for a word describing "love and affection shared by two straight males." The game show was looking for "bromance." But Watson's somewhat befuddled response was "relationship," which Gondek called "kind of sweet, really." And the computer was inadvertently funny responding to the clue, "Give a Brit a tinkle when you get to town and you've done this." Watson deadpanned: "Urinate."

Gondek said that IBM also had to find a way to program Watson to make a connection between two apparently unrelated things, coming up with an answer in two words that also had to rhyme, as in, A.: "A long, tiresome speech delivered by a frothy pie topping." Q: "What is meringue-harangue?" In one rhyming test that the computer flunked, the clue was a "boxing term for a hit below the belt." The correct phrase was "low blow," but Watson's puzzling response was "wang bang." "He invented that," said Gondek, noting that nowhere among the tens of millions of words and phrases that had been loaded into the computer's memory did "wang bang" appear.

To create and program Watson, IBM collaborated with researchers at eight U.S. universities, including MIT and the University of Massachusetts Amherst. The main contribution of the UMass investigators was to work on information retrieval, or text search, a key aspect of the "DeepQA" technology that enables computer systems to provide answers to questions in natural language over a broad range of knowledge. Researchers at UMass helped IBM develop a means for the computer to take a first step searching for an answer by retrieving text that is most likely to contain accurate answers.


The role of the team in Cambridge, led by Boriz Katz of MIT's Computer Science and Artificial Intelligence Laboratory, was to help Watson take a second step by breaking down questions into simple sub-questions for responses that can be rapidly collected and then recombined into likely answers. The MIT researchers pioneered an online natural language question answering system called START.

Gondek said that while human "Jeopardy!" players might press their buzzers "on instinct" if they think they know an answer, Watson will not buzz in (unless it is losing so badly it's "desperate") until it thinks it has a high probability of having the right answer. In the current match, its calculations for three possible answers are shown in a graphic beneath its avatar. But it is not always right. In the initial round, it gave a wrong answer that it figured had more than a 90 percent chance of being right. Gondek said it is also "deaf and blind," which is why no audio-video clues are being used. That may be the reason Watson repeated a wrong answer that Jennings had just given--it can't hear the other competitors. (Like its human opponents, Watson also is not connected to the Internet during the match).

Another thing Watson cannot do is try to buzz in by listening closely to host Alex Trebek and anticipating when he'll finish reading a clue, down to a fraction of a second. Erin McClean, a Boston University undergraduate who won the 2010 Jeopardy! College Championship, told the MIT crowd that "I went off of Alex's voice. It worked because I had practiced so much at home, so I got into a rhythm." By contrast, Watson has to rely on a set of lights that come on as soon as Trebek finishes.


McClean described her winning technique as a combination of virtually obsessive practice and savvy use of the internet to learn from tips offered by past contestants. She recommended that anyone who aspires to be a contestant go to two websites: http://www.pisspoor.com/jep.html and http://www.j-archive.com/. The first website, "How to Win on Jeopardy!," was created by two-time champion Karl Coryat. The second one, "J!Archive," says it is a fan-created site with "203,444 clues and counting."

Gondek said that win or lose, IBM hopes to use the technology it has developed for Watson in various product applications, especially in health care. "I can see how Watson could function as a physician's assistant," he said, compiling information to help a doctor with a diagnosis." While MIT students and staff may be rooting for Watson, given the university's role in its development, McClean said, "I'm rooting for the humans. I really think they can do it."

February 16 Update: : After the first of two games in this match, Watson has what may be an insurmountable lead, giving its two human opponents a shellacking, to borrow a word from President Obama's lexicon. With one more game to be played Wednesday, the supercomputer has $35,734 to $10,400 for Rutter and $4,800 for Jennings.

At this point, the only hope for the humans may be that Watson gets reckless and blows its wad, although that didn't happen in the first game, even though the computer's answer in Final Jeopardy was wrong. In the category "U.S. Cities," in response to the question, "Its largest airport was named for a World War ll hero; its second largest for a World War ll battle," Watson said "What is Toronto?????," with the extra question marks showing its uncertainty. (For good reason--it was not the right city and not even the right country). Both Jennings and Rutter got the correct answer: "What is Chicago?"

On the blog "Building a Smarter Planet" (http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-day-two-the-co...), David Ferrucci, manager of the Watson project, explained why the computer faltered on the final clue. But Ferrucci was encouraged by the fact that Watson, recognizing that that it wasn't sure about the answer, bet very cautiously, risking only $947. "That's smart. You're in the middle of the contest. Hold onto your money. Why take a risk?," he said.

That doesn't seem to bode well for Jennings and Rutter.

Tournament Finale Update: It wasn't even close in the end. Watson crushed its flesh-and-blood opponents with a two-day total of $77,147 to win the tournament. Jennings was a distant second with $24,000 overall, with Rutter in third place with $21,600. But the second game was much closer than the first, and with a little luck, such as hitting both Daily Doubles, Jennings might have been in a position to win it, salvaging at least one game for the humans.

Going into Final Jeopardy, where the category was "19th Century Authors," Watson was ahead with $23,440, but Jennings was not that far behind with $18,200 and Rutter in third place with $5,600. The clue was "William Wilkinson's 'An Account of the Principalities of Wallachia and Moldavia' inspired this author's most famous novel." All three contestants correctly answered "Who is Bram Stoker?" (the author of "Dracula"). But while Rutter bet everything he had, Jennings bet only $1,000. Unlike the previous night, when Watson flubbed the Final Jeopardy clue, but lost less than $1,000 with a cautious bet, the brainiac computer this time obviously felt confident. It bet $17,973, winning the game with $41,413.

Jennings accepted the result with equanimity and perhaps, resignation. Echoing a line from an episode of "The Simpsons" he wrote on his video screen, "I for one welcome our new computer overlords."