Thoughts on Watson: the Jeopardy robot Part III

OK. Watson won. He won kinda big time, and I was gonna post a picture of a chain-link-fence-holding Sarah Conner engulfed in flames on judgement day, but it was sort of gruesome and anyway I don't really feel that way about this particular robot triumph. That said, burying an arms cache in the Mexican desert and maybe learning a few tips on how to outsmart logic-beholden machines via a re-read of Asimov's I Robot might not be a terrible idea. (no, watching the movie won't help)

Before we get into the gloomy excitement of computers getting really really smart all of a sudden,  let's quickly discuss the saving grace for humanity here which is: Ken Jennings managed a (reasonably funny and certainly appropriate) Simpsons reference within his final jeopardy answer!

We're still good at something! Being funny! Take that, machines!

In case you're not a Simpsons nerd, see the classic Kent Brockman clip below.



Maybe IBM's next challenge should be to develop a robot that wins Last Comic Standing. No, seriously, give it a shot, IBM. It's even OK if it uses props, or ventriloquism, or a redneck-y catch phrase, but I think your best bet is to have it write 45 minutes about the quirks of being robotic, then stock the audience with robots. But perhaps I digress.

Back to the show.  What can I say except to ask more questions about how Watson works? I find this stuff sort of endlessly fascinating.

Why does it seem like Watson's so much better at Double Jeopardy than Single?

Does Watson benefit from any sort of momentum of confidence factor after a series of correct answers (like a human would)? (Presumably not...so, followup question: Would that be an inherently bad thing to build into a computer's programming? Discuss)


How did Watson put together Moldavia and Wallachia to get Bram Stoker but not get the Chicago airport question?


Let me jump into that one.


As a reasonably intelligent human being, I gauged both of these (final jeopardy) questions as fairly easy, although I would agree that the Chicago one was a little tougher. Although the inherent difficulty of any question is debatable and of course skewed by whether someone knows or does not know the answer, I feel my opinion is at least somewhat founded in objective analysis and also supported anecdotally by the fact that both human players got both questions correct.

I'm being presumptuous about how Watson thinks through these questions, but what the hell (bbq). To get to last night's final Jeopardy solution (clue paraphrase: "So and so's published anthropological survey of Moldavia and Wallachia was the inspiration for this author's most famous work"), I feel Watson must have had to throw "Moldavia" and "Wallachia" into the gears and realize (quickly) they were a reference to Romania. (Wikipedia informs me that these comprise the north and south historical regions of what is now modern day Romania). From there the path gets a little murky though. The category was "19th Century Authors". So now Watson must filter through a list of authors with whom Romania is associated?  Or does he, in his geographical search come up with keyword "Transylvania"...which when filtered through the category title yields "Bram Stoker", author of Dracula. Perhaps "Dracula" has to occur first to Watson but I sort of doubt it. This all seems pretty reasonable, except if this type of database list generation and subsequent list cross referencing is how Watson arrives at answers...
then why not nail that Chicago airport one?

Pull up a list of major airports, filter by US cities (or don't, even) and cross reference with cities with 2 airports. Even without the US filter (or if the US/American filter includes all of the cities in the Americas) this ought to yield a fairly short list.  Then filter by historical names, and see which are associated with WWII.  As I mentioned, Toronto's airport is Lester Pearson International (YYZ), but he's got no direct association with WWII (that I can see). A good but incorrect guess would have New York City (because of JFK)
I find all this puzzling, is all. Hopefully we'll get more coverage and learn more about how Watson ticks in the near future.  For now I'll take our triumphs where we can get them, but more pertinently...where's my holodeck?

Thoughts on Watson: the Jeopardy robot Part II

Last night was sort of hard to watch, right? Watson mopped the floor with the humans and went out to a $20,000 lead. Last night Watson got both Daily Doubles and eventually played in Final Jeopardy. The main question I want to ask is:

How does Watson decide what to wager?


Presumably it's an algorithm based on a simple (quick) examination of Watson's stores of info on any given category, crossed somehow with the game's current scores and score gaps, etc... Does Watson (actively) take into account the remaining board squares' monetary values when deciding what to wager? Does Watson learn about playing styles as the game goes on? In other words, does Watson know he's winning and is likely to continue to win, or only that he's currently ahead in points?

What jumps out to me most about last night was the final Jeopardy clue.  The category was US Cities, and the clue had to do with airport names.  Paraphrase is "what US city's largest airport is named for a WWII hero and second largest named for a WWII battle?" Ken, Brad, and I came up with the correct answer as 'Chicago' (references to airports O'Hare and Midway) but Watson flubbed it pretty big with "Toronto".

For starters, Toronto's not in the United States....so it seems genuinely strange to me that Watson could try to pass that one off as correct, even as a shaky guess. Secondly, Watson only wagered $947 bucks on the clue.  One obvious explanation of that could be that Watson was way way ahead in the score, so oftentimes players in that position won't jeopardize (ha!) the guaranteed win in a gambit to accumulate money...however those helping normal human contestants give them a series of scenarios (at least, this is my understanding as a viewer) to help them decide how much to wager.

Watson was almost $20,000 dollars ahead, so he could have bet almost 20 times as much and still ensured his overall win in this round. Watson did no such thing, at least in part because the two day game is cumulative and so he must have it in his programming to generate as much money as possible...

This is all a roundabout way to get back to what made the category "US cities" appear daunting to Watson, and why he guessed incorrectly on a clue that at least three humans got easily. Why be intimidated by this category, and then furthermore why get it so wrong? It seems a cursory cross reference of major US airport names with WWII associated people would have quickly yielded the Chicago answer.  Also note that Toronto's largest airport (YYZ, the one Rush wrote the song about) is called Lester Pearson Airport after a Canadian prime minister.

More questions than answers...looking forward to the conclusion.

"too clever is stupid, dude."
 - Icepick (from Skate or Die II)

Thoughts on Watson: the Jeopardy robot Part I

Last night was part 1 of 3 for the competition between IBM's Watson and the two Jeopardy wizards only as known as Ken Jennings and Brad Rutter. I've been anticipating it since I first heard it was happening late last year, but I gotta admit I hadn't really thought that hard about it beyond a standard "holy shit' feeling.



I've long been a proponent of Star Trek style technology, despite it's inherent evilness and capability to trap us inside Agatha Christie trains or make us obsessed with gaming technology. I've publicly stated that I'll have the matrix plug installed as soon as it becomes available (read: covered by insurance) because once that thing exists, who knows what's what in any real sense so why not jump on the jelly wagon post haste and ride it to freedom (or slavery). Watson appears to be an important step in this lineage.

Watson takes the questions and spits out answers in plain (question-oriented) english after ringing in on the buzzer. I had been under the impression it would be analyzing Alex's actual speech in order to receive the clue info, but apparently that's not the case.  Trebek said Watson receives the clue as a text file simultaneously to it being revealed to the player. That's slightly less impressive--(but still cool) and it brings up a couple questions that weren't answered last night:

Does Watson get the text file entered at the moment the question gets "revealed" on TV (and presumably to the contestants as well) or immediately after Alex is finished reading it?

My guess is the former, which slants the game significantly in Watson's favor, since it's reading speed has got to be worlds faster than any human. In reality, the time it takes Alex to read through a clue is almost an eternity for a machine so gosh darn sophisticated.  But OK, still pretty cool.

Here's another relevant question though:

How does Watson receive the category information?


This is more vague, I guess. Watson presumably receives the categories as text files at the same time as contestants, but this point was not specified. It's important, I think, because as soon as you hear the categories your (human) brain is priming it's internal filing cabinet for what might come up in the context of said category. So assuming Watson gets them simultaneously with the humans. I wonder how the programming parsed that information specifically, or if Watson chooses how to move through the board based in part on a perceived wealth or paucity of information in any given category. Maybe tonight's broadcast will reveal more of this type of stuff.

A couple other observations I'm hoping to explore after seeing more of the game:

I noticed that in one instance, Ken Jennings answered something incorrectly (in the "what decade?" category) and then that Watson rang in and gave the same (incorrect) answer directly afterward. Did Watson's developers miss a fairly obvious aspect of the game with this, or was this instead an extremely difficult programming obstacle?  Or perhaps they are scrambling as I type to fix this glitch.

The visualization of Watson's thought process was pretty intriguing.  The concept of a confidence threshold was covered and then presented as a bar graph on the screen. Although Watson was typically quite confident in the answer (and subsequently correct) there were a few instances where Watson wasn't particularly certain which of three top answers to select. Here's one thing it made me wonder:

Does Watson's option weighting and/or overall confidence in any given answer correlate to what us humans might call "the overall difficulty" of the question? 

Although Jeopardy works hard to weight the difficulty of the clues accurately with a monetary value, it's easy to agree that this is a flawed system. What makes question difficult? (besides not knowing the answer).
Example: In the "Literary Villains APB" category, the mid-level clue was indicating the villain from the Harry Potter series: Voldemort.  Watson incorrectly surmised "Harry Potter" and the confidence bars indicated a degree of confusion (although the threshold was not met). Two pieces of information in regards to that are worth thinking about: 1. the clue didn't mention "Harry Potter"  and 2. "Voldemort" is (somewhat) rarely referred to by his actual name in this series of books.
Watson must be using a sort of "fill in the blank" algorithm which correctly identified other words in the clue (like Hogwarts) as being related to Harry Potter, while also correctly identifying that "Harry Potter: himself did not appear in the clue. This may be an interesting, somewhat non-intuitive piece of the "what makes a clue difficult" puzzle.
The second point however, speaks more directly to what it means for a question to be difficult. There is simply less information connecting that particular answer to the clue, less repetition of the specific idea, fewer linguistic bridges.

I'd like to get more into this avenue of thought after I watch Double Jeopardy tonight. Right now, Watson is tied for the lead at $5000.  Humanity still has hope, and the Holodeck may still be a ways off I guess.

Note: wanted to get my thoughts down first, without reading others' take...but now that I did here's another article, which makes mention of an interesting contextual aspect of correctly answering that I didn't recount above--very interesting stuff...referred to as "leg-gate" in the comments.

Chinese vs Japanese / Man vs Robot

OK, first the bad news. I re-watched The Matrix: Reloaded. When the original Matrix came out I was a sophomore in college and I went to the Newark shopping center to watch it with my bros. The ad campaign for the film had been a really good example of creating actual intrigue in regards to a blockbuster movie with the whole "What is the matrix" tagline. Anyway we were stoked to finally see it and the movie wrecked shop on our 19-year old brains and didn't disappoint at all.

Of course years later when the 'trilogy continued' I was only cautiously stoked (at best)...but paid up to see more slo-mo ass kicking and 360 bullet dodging. Everyone knows it sort of sucked...at least relative to the original film, and I hadn't forgotten this but the other day I was in a position to pick out a mindless movie to watch while I was completely immobilized for about 3.5 hours doing leukophoresis (white blood cell donation) and so I was thinking "why not? this movie had some cool fights in it and I like those". And to be fair, The Matrix: Reloaded held my complete attention while my entire blood volume was drained, filtered, and then put back into my bod on the other side.

The Matrix films are basically very high concept/high budget martial arts movies, so on to the kicking people in the face part. The coolest fight in the movie is probably the one where Neo fights like hundreds of Elronds all dressed up for the white house press correspondents' dinner.

I realize that Neo is supposed to be seeing everything in slow motion, but he also kind of kicks in slow motion, which isn't really so cool. Anyway--like lots of kung-fu movies of yore, this one employs lots of physically impossible wire trick stuff but also makes copious use of that patented (they wish) slow down and strafe camera work. Difference is, this isn't even trying to look particularly real, which works great for Keanu. When he gets the pole out and goes all tether ball it's pretty awesome regardless.





Diagnosis:
This movie makes very little sense, and when I saw it the first time I recall thinking: it's a windup for the next movie...hopefully 'Martix: Revolutions' will swing hard and hit it out of the park. We all know that didn't pan out (maybe I'll go ahead and rewatch that one soonish as well), so this movie just seems jumbled up unnecessarily. I think it would have worked better if they dropped a little of the over-reaching hero quest narrative and focused on what I'm seeing as the primary (read: most interesting) plot driver of the movie which is the following: given that this thing called the matrix exists, check out all the weird alt-programs that float around inside of it (including, as the climax indicates, Neo himself).

Perhaps the most misguided/mistakenly hilarious scene of all is the notorious "cave-rave" pictured above. The part that really tickled me was right before the Zion-techno-oontz starts bumping where Morpheus announces (triumphantly) 'let us shake this cave' in righteous defiance of the robot overlords that will soon consume and enslave us. Youtube wouldn't let me embed this one (the line is at about 1:30)

http://www.youtube.com/watch?v=V5ZWNOm4r2k&feature=related

So amongst such original ideas as twin albino badasses, orgasm cakes, keymasters who unlock secret stuff (wait a sec...), and having Harold Perrineau being able to steer a ship straight and true despite being in a weird non/ultra reality, this movie is probably worth about:

.56 of the 1

sidenote for any science-types reading this: when Neo 'reaches inside' Trinity to kickstart her heart Mola Ram-style at the end, all I could think was "terrible sterile technique, dude."

1/24/11 topical update: Keanu, please don't make anymore of these.

--

I also watched a modern kung-fu film called Ip Man.  It's on Netflix streaming and you should watch it. It's much more of a "film" film than most martial arts movies, but the fighting in it is really really kickass. (see below)

This may be old news to some of you because I believe they already released a sequel to this bad boy...but in quick summary, Ip Man is a real guy who lived in southern China during the second Sino-Japanese war, and was the first to teach Wing Chun style kung fu. He had tons of students the most famous of whom is Bruce Lee.



The movie is set during a time of Japanese occupation, and Ip Man is the old standby peaceful man character who doesn't really want to fight, but holy crap look out when his blood gets angried up.

Though this scene below is only about half way through the movie...it's probably the coolest fight scene.  The setup: Japanese soldiers are challenging all the Chinese guys to come fight them, to prove that Japanese martial arts is better or whatever, and because they're an occupying army they're bullying them into it big time, and being dicks in general. Ip Man's all like "I choose not to fight" etc.. and that's part of how you know how unbelievably badass he is, because he's super calm about it and everything. But then...well, you know.



Were you digging all the multiple rapid punches and painful limb breaks like I was?
You should watch this movie for more of the same. Netflix streaming, like I mentioned. Do it.

9 out of 10 japanese guys with broken legs and multiple face bruises agree.

Year in Review

So what did I do this year? Let's recap in a slapdash way that romanticizes minor occurences while downplaying major ones and adds a sheen of coolness to basically everything.

I started 2010 watching fireworks on a couch in Kuala Lumpur, and ended it just now at a karaoke bar somewhere in the LA outskirts belting out "Alive" with two Japanese dudes I just met. Pretty solid on both counts.

I played the last show with an old band and the first show with a new one. I took two GREs. I got engaged. I soaked in a Japanese mountain hot spring and walked on the great wall. I turned 30.

I saw Arcade Fire in Berkeley, and a Broadway production of South Pacific. I went to the symphony a couple of times. I bought Guess Who and Dvorak records. I ate a fried Oreo.

I drank some wine from 1910 and made eggnog from scratch. I watched The Sound of Music for the first time, and ran into Tom Waits at the Hotel Utah in SF. I had a surprise birthday party and was completely surprised.

What comes next?