Update on Watson vs Jeopardy

[Elomin Sha]

http://gamepolitics.com/2011/01/14/ibm- ... y-test-run

Watson has trounced his human competitors in a test-run.

[Myrph]

http://gamepolitics.com/2011/01/14/ibm-computer-trounces-humans-jeopardy-test-run

Article is quite interesting, once you can get to it, though doesn't say too much new stuff.

The fact that we've managed to develop a computer that can analyse the subtle nuances in the English language is quite incredible, so it should be interesting to see the actual event, when it happens.

[empath]

Good, this means that in maybe twenty years time, we might have voice-recognition systems on automated phone menus that actually understand the difference between "get me an operator, please" and "upgrade my services to a higher monthly fee, please".

Maybe.

[Elomin Sha]

Stupid edit, I repasted the link so it shouldn't have broken it. Grrrrrrrr.

[xdeathknightx]

I did read somewhere that his win was in part to be contributed to the fact that the machine can buzz without delay. And from what I saw the scores weren't that far apart so it wasn't a huge trump/

[Kaiserin]

Everything else aside, the ability to buzz without delay would be a significant advantage, especially over the course of a game. How often do you see all three contestant mashing their buzzers at the same time.

[Master Gunner]

Well with the games done, I figured I should resurrect this thread to reflect on it.

Watson did far better than I expected, though when it did make a mistake, it was odd (chiefly Toronto being a US city, I'd be interested in seeing the debug logs from the queries which resulted in that response). All in all, this certainly represents large improvements in computing. By the end of the decade, it's looking like our computers will be able to give the Enterprise-D's a run for its money.

One thing that I noticed is that, dispite being a programmer, I was rooting against Watson. Maybe I just fear the singularity? Combine Watson with some of the other current AI projects out there, and it won't be long before they can start writing and improving their own code (not to mention the code for any company I could work for).

[tak197]

I think we were all rooting both for and against Watson at different points. They did answer our questions about how it would actually compete (namely having to still manually ring in), and all in all, they proved that, like MG said, technology and computing has come a long way since the Deep Blue challenge.

What I'm interested in is the logic functionality it used for wagering. They were very specific wagers, so I'm thinking it has some super-accurate risk/reward program that can figure out how likely it will get the answer right without knowing the actual question.

[BillTheCat]

Watson did far better than I expected, though when it did make a mistake, it was odd (chiefly Toronto being a US city, I'd be interested in seeing the debug logs from the queries which resulted in that response).

From http://arstechnica.com/media/news/2011/02/creators-watson-has-no-speed-advantage-as-it-crushes-humans-in-jeopardy.ars
The answer it messed up on was "Its largest airport was named for a World War II hero; its second largest, for a World War II battle." in the topic "US Cities."

Welty said Watson was also confused by the second clause of the answer, "its second largest, for a World War II battle," because it can't understand context. It's obvious to humans based on phrasing that the second part of the answer is also referring to an airport, but Watson is unable to parse implicit information.

They also state in the article that Watson doesn't rely on the category to determine the type of question needed so that is why it didn't strictly limit itself to only US cities.

Edit: In the article they also mention that they had a game theorist come up with a formula to calculate the optimal amount to bet on daily doubles and final jeopardy.

[MattAn]

I liked the part where Watson told Andy Richter that he slept with his wife and she had.. Ahem.. "An ass he could bounce a coin off". Then Andy bashed him with a baseball bat. Repeatedly.

...Oh, we're talking about the Jeopardy Watson? I like CONAN's better.

[Jillers]

You know what I was thinking they could use Watson for? New text-based adventure games. What an amazing text parser.

[empath]

A truly DYNAMIC game, even - I'd say it's got the horsepower to keep track of all the players' changes and actions.

...heck, maybe even a multiplayer text adventure with such a realistic and flexible parser...or IOW, a MUD that is its own admin?

[Jillers]

See, all that I was thinking of was how it'd be great to e able to type in "Get pot" and not have the computer reply with:
"I don't understand get" or
"I don't understand pot"

[gcninja]

wouldnt it be

Code: Select all

sudo apt-get pot

?

[Mercenary]

The interesting thing about Watson buzzing immediatly is kind of a false. Depending on his confidence level and buzz threshold he'll take a couple of seconds.

http://www-943.ibm.com/innovation/us/watson/what-is-watson/the-future-of-watson.html

link has a video that at one point shows how Watson parsed the airport question. In which the two humans answered Chicago and Watson answered "Toronto????"



Anyways to take a line from Ken I believe:

I for one welcome our new computer overlords

[GaProgMan]

wouldnt it be

Code: Select all

sudo apt-get pot

?

Precisely.

[Dubious_wolf]

"I for one accept our new computer over lords.".

[Myrph]

"I for one accept our new computer over lords.".

I'm not sure it would be particularly easy to play cricket on a laptop, but you're more than welcome to give it a go! I'll go wicket keeper!

[tak197]

http://www-943.ibm.com/innovation/us/watson/what-is-watson/the-future-of-watson.html

link has a video that at one point shows how Watson parsed the airport question. In which the two humans answered Chicago and Watson answered "Toronto????"

Yeah, apparently there is a Toronto in Kansas that has two airports. It just didn't take into account the airport names.

[Evil Jim]

I went out of my way to see these episodes of Jeopardy & was quite entertained.

I was a little disappointed however, to learn that Watson did not have a camera or microphone to read the clues on the monitors or listen to Alex as he read them. Instead, they just gave Watson a text file of the clue & let him figure it out from there. That seems a bit primitive considering what voice recognition software can already do these days. It's too bad the show writers didn't throw in any musical clues like they do from time to time, or show a photo of a landmark & make the contestants come up with the year in which it was dedicated, as is another type of clue. As it was, the format was already slanted in Watson's favour.

I wonder why Ken didn't bet it all on Final Jeopardy the final day. I don't remember the scores but I think he might have won if he had. He said "all or nothing" was the way to go at an earlier Double Jeopardy, so you'd think he would have bet the wad, especially considering he would be leaving with $100,000 minimum no matter what happened.

[theDreamer]

The point of the game wasn't to prove anything.

It was a test.

A test of Watson's ability to parse language.

A camera would have taken all of milliseconds longer to parse, he still would have been done in no time at all.

And audio recognition is known to fail frequently. Sure, it might have been possible, but then you're testing not only Watson's ability to understand language, but his ability to comprehend audio.

And that is bad science.

[Master Gunner]

Indeed. In some implementations of Watson's software in the future, it may be combined with other systems to allow it to understand a wider variety of data, but that's outside the realms of this test. All speech-to-text software does is, well, convert sounds to text strings, which could then be fed into Watson exactly as they were during the games. As they were not working on designing/implementing revolutionary speech recognition programs (which are considerably easier to test), it would have being an uncontrolled variable and invalidated the entire test.

Eventually I'm sure there will be a Jeopardy game featuring a computer contestant that can hear and see, but that will be a much later test.


As for Ken's Final Jeopardy bet: if he had bet everything, he still wouldn't be able to come close to beating Watson's two-day total (and Watson would know that, and calculated his bet accordingly to make sure that it wouldn't be a possibility). So, he bet to maximize his odds of coming in second.

[Jillers]

wouldnt it be

Code: Select all

sudo apt-get pot

?

I have no idea what you mean.

[Lord Chrusher]

It is a Linux joke.

apt-get is a program for downloading and installing software. If you wanted to download and then install the program 'pot' you would type

Code: Select all

sudo apt-get install pot

The sudo command executes the apt-get command as root (the admin user).

[Jillers]

Ah... riiiiiighhhhhttt.... I'm going to go over here now.