Also there's the problem of context. When someone asks you a question you generally know what kind of answer they expect.
In this classic example from Richard Feynman [1] the answer to "Why is Aunt Minnie in the hospital" can be "She broke her hip", "She slipped on the ice and broke her hip", "An ambulance took her there", "Because her husband is concerned for her well being".
What's interesting about the Watson approach is that, by taking intelligently composed web pages as input (as opposed to randomly generated texts, or an arbitrary list of atomic facts) the context is already "baked in". Put another way, people make web pages with sentences like "Aunt Minnie is in the hospital because she fell and broke her hip", and don't make web pages with sentences like "Aunt Minnie is in the hospital because the ambulance was headed there and the friction between Aunt Minnie and the bed in the ambulance was high enough to keep them from becoming relatively displaced".
The answer from an intelligent alien who was context-naive might be something like so:
Aunt Minnie is a human being living in the 21st century Western culture of Earth. When humans in this region and time period become sick or injured, relatives and friends, or medical workers working on their behalf, will transport them to a hospital for treatment. Aunt Minnie was injured on a sidewalk made slippery by seasonal ice. Such injuries aren't uncommon but are treatable with various surgical and therapeutic techniques. This demonstrates the humans vulnerability and heralds our inevitable victory to harvest them as a source of food.
It is indeed the appropriate answer here, that seems pretty clear to me. What's not clear is how we both came to the same (or similar) conception of what's the right answer from all those possible, arguably correct, answers.
This is basically a restatement of the Frame Problem, which I believe is an unsolvable problem if you limit yourself to software. This is basically my PhD, and I only reached this opinion after much wailing and gnashing of teeth.
In some sense this program isn't limited entirely to software, but extends to the full gamut of cultural human lived experience, by exploiting the intelligence already present in the input corpus.
Yes that's a hard problem but thankfully an automated system would get a pass on that, i.e., for the problems we are trying to solve, any of these answers would be considered a reasonable answer... because it's hard enough to even get to a reasonable answer.
Question answering is actually much harder than was is presented in the article because most questions people ask are not factoid questions. Consider the following that Watson needed to address at a recent customer engagement (an insurance company):
* Can you help me with life insurance?
* What benefits will I lose when I leave?
* What last minute things should I take advantage of in the military before I go?
The answers to these questions are not entities (people, place, dates) as in factoid questions. They may be procedures, they may depend on the user, further clarification/dialog may be required before being answered, they may not have a single answer or require a combination of answers, etc.
Also Watson does not try to formalize all knowledge even when answering factoid questions. Many of the candidates generated do not actually come from a formal knowledge graph, instead they are generated on the fly from analyzing all the pieces of text that seem contextually relevant.
There isn't that much publicly available information about how Watson works - I'm pretty sure I'd read everything there was in January this year. There might be some more now?
I can't even imagine where to start with "Can you help me with life insurance" - beyond maybe triggering standard sales pitches.
How does Watson decide when to use the knowledge graph and when to look more broadly?
The right way to answer "Can you help me with life insurance" is to start asking the user questions to learn more about their needs - that's one of the way we are currently trying to go beyond the Jeopardy use case.
I'm yet to see a good public implementation of conversational question answering. It's pretty clear Google is headed down that path with Google Now, and I guess Siri is going the same way.
Traditionally question refinement is done using filters and clustering, but of course this focuses the result sets, and is the opposite of what you need in conversational question answering.
Question answering is an amazing problem. I built factoid QA systems using a mix of NLP, open relation extraction (dependency relation classification via SVMs), semantic web stuff via triple stores (virtuoso) and answer candidate ranking algorithms (classify the questions in to different types via keywords) and using those to determine whether I should use a search engine, or lookup system.
I think for those who want to get started in the field understanding some of the more complex NLP techniques, information retrieval, and some idea of how triple stores work gets you to building a QA system.
This is actually why I jumped in to deep learning. The hardest part of NLP for open ended question answering is coming up with all of the different features.
Deep Learning with textual representations fixes this.
The tech stack was http://uima.apache.org for those who are curious. I'd be happy to answer anything specific. QA systems are a little esoteric due to the combination of techniques used to build them, but they aren't as mysterious if you put in the effort to understand what they actually are.
A couple of years ago I wrote a question answering system too; it's just sitting on my drive. I haven't decided what to do with it. Maybe open source it?
In this classic example from Richard Feynman [1] the answer to "Why is Aunt Minnie in the hospital" can be "She broke her hip", "She slipped on the ice and broke her hip", "An ambulance took her there", "Because her husband is concerned for her well being".
What's interesting about the Watson approach is that, by taking intelligently composed web pages as input (as opposed to randomly generated texts, or an arbitrary list of atomic facts) the context is already "baked in". Put another way, people make web pages with sentences like "Aunt Minnie is in the hospital because she fell and broke her hip", and don't make web pages with sentences like "Aunt Minnie is in the hospital because the ambulance was headed there and the friction between Aunt Minnie and the bed in the ambulance was high enough to keep them from becoming relatively displaced".
[1] https://www.youtube.com/watch?v=MO0r930Sn_8
(Edit: I notice, of course, that I have now created a web page with that exact sentence.)