http://thesocialweb2013.wordpress.com/2013/03/11/guest-lecture-by-chris-welty-ibm-research-what-watson-tell-us-about-cognitive-computing/ Watson is a computer system capable of answering rich natural language questions and estimating its confidence in those answers at a level rivaling the best humans at the task. On Feb 14-16, 2011, in an historic event, Watson triumphed over the best Jeopardy! players of all time. This success was important in numerous ways, one of which is as a prominent exemplar of a new generation of computing systems, that we now call Cognitive Computing Systems. Cognitive computing involves a new synthesis of classic AI problems such as language understanding, image and video processing, with big data, human computing, and massive processing. Social systems are an important driver of cognitive computing, as they provide data used by many systems (dbpedia, freebase, twitter) as well as numerous potential applications. In this talk I will discuss how Watson works, and cognitive computing with an eye towards social computing.

1. What Watson tells us about
Cognitive Computing
Chris Welty
IBM Research
ibmwatson.com
Do Not Record. Do Not Distribute.
© 2011 IBM Corporation

2. What is Watson?
§ Open Domain Question-Answering Machine
§ Given
– Rich Natural Language Questions
– Over a Broad Domain of Knowledge
§ Delivers
– Precise Answers: Determine what is being asked & give precise response
– Accurate Confidences: Determine likelihood answer is correct
– Consumable Justifications: Explain why the answer is right
– Fast Response Time: Precision & Confidence in <3 seconds
– At the level of human experts
– Proved its mettle in a televised match
– Won a 2-game Jeopardy match against
the all-time winners
– viewed by over 50,000,000
2 © 2011 IBM Corporation

3. What is Jeopardy?
§ Jeopardy! is an American quiz
show
– 1964 – Today
– Household name in U.S.
§ answer-and-question format
– contestants are presented with
clues in the form of answers
– must phrase their responses in
question form.
– Open domain trivia questions,
speed is a big factor
§ Example
– Category: General Science
– Clue: When hit by electrons, a
phosphor gives off electromagnetic
energy in this form
– Answer: What is light?
© 2011 IBM Corporation

4. What is Cognitive Computing?
§ Increasingly, machines are being asked to add their computational
power to problems which are not inherently solvable
§ Traditionally, these problems came from AI
– The hardest AI problems are the easiest for human intelligence:
vision, speech, natural language – these are not actually associated
with “being intelligent”
– Human intelligence provides solutions, but does not scale
§ Cognitive Computing is founded on four principles
Learn & improve. Cognitive computing systems Assist & augment human cognition. Cognitive
focus on inexact solutions to unsolvable problems computing addresses problems that lie squarely in
that utilize machine learning and improve over time. the province of human intelligence, but where we
Often they combine multiple approaches and must can't handle the volume of information, penetrate the
integrate them effectively. They must learn from complexity or otherwise extend our reach
humans, in more and more seamless ways. (physically).
Interact in a natural way. Cognitive computing
Speed&Scale. Cognitive computing harnesses the
provides technologies that support a higher level of
clear advantage machines have over humans in
human cognition by adapting to human approaches
their ability to perform mundane tasks of arbitrary
and interfaces...over the next several decades it will
complexity repeatedly, whether it is the scale of the
incorporate essentially all the ways humans sense
data or the complexity of the task.
and interact.
© 2011 IBM Corporation

5. Examples of Cognitive Computing
§ Web Search
§ Image Search
§ Event Search
§ Social Computing
§ Natural Language Processing
© 2011 IBM Corporation

6. The Jeopardy! Challenge
Hard for humans, hard for machines
$200 $1000
Broad/Open If you are looking at The first person
Domain the wainscoating,for different reasons.by name in
But hard mentioned
you are looking in ‘The Man in the Iron
this direction. Mask’ is this hero of a
Complex previous book by the
Language Who is
same author.
What is down? D’Artagnan?
High For people, the challenge is knowing the answer
Precision
For machines, the challenge is understanding the
question
Accurate
Confidence $600 $800
In cell division, mitosis
The conspirators against
splits the nucleus &
this man were wounded by
High cytokinesis splits this
each other while they
Speed What is
liquid cushioning the Who is Julius
stabbed at him
nucleus
cytoplasm? Caesar?
6 © 2011 IBM Corporation

7. The Winner’s Cloud
What It Takes to compete against Top Human Jeopardy! Players
Each dot – actual historical human Jeopardy! games
Top human
players are
remarkably
good.
Winning Human
Performance
Grand Champion
Human Performance
2007 QA Computer System
More Confident Less Confident
© 2011 IBM Corporation

8. The Winner’s Cloud
What It Takes to compete against Top Human Jeopardy! Players
Each dot – actual historical human Jeopardy! games
Winning Human In 2007, we committed to
Performance making a Huge Leap!
Grand Champion
Human Performance
Computers?
2007 QA Computer System
Not So Good.
More Confident Less Confident
© 2011 IBM Corporation

9. DeepQA: The Technology Behind Watson
An example of a new software paradigm
DeepQA generates and scores many hypotheses using an extensible collection of
Natural Language Processing, Machine Learning and Reasoning Algorithms.
These gather and weigh evidence over both unstructured and structured content to
determine the answer with the best confidence.
Learned Models
help combine and
weigh the Evidence
Evidence
Sources
Answer Models Models
Sources Deep
Question Answer Evidence Models Models
Evidence
Scoring Retrieval
Primary Candidate Scoring
Answer Models Models
Search
Generation
Question & Final Confidence
Question Hypothesis Hypothesis and
Topic Synthesis Merging &
Decomposition Generation Evidence Scoring
Analysis Ranking
Hypothesis Hypothesis and Evidence
Generation Scoring Answer &
Confidence
... © 2011 IBM Corporation

10. Example Question
Keywords: 1894, C.W. Post, Related Content
In 1894 C.W. Post created … (Structured & Unstructured)
created his warm Lexical AnswerType:
(Michingan city)
cereal drink Postum in Date(1894)
this Michigan city Primary
Question Relations:
Search
Analysis Create(Post, cereal drink)
…
Candidate Answer Generation
General Foods [0.58 0 -1.3 … 0.97]
1985 [0.71 1 13.4 … 0.72]
Post Foods [0.12 0 2.0 … 0.40]
1) Battle Creek (0.85)
aramour
Battle Creek [0.84 1 10.6 … 0.21]
2) Post Foods ( 0.20)
Grand Rapids [0.33 0 6.3 … 0.83] 3) 1985 (0.05)
… [0.21 1 11.1 … 0.92]
[0.91 0 -8.2 … 0.61] Merging &
… Ranking
… [0.91 0 -1.7 … 0.60]
Evidence
Retrieval Evidence
© 2011 IBM Corporation
Scoring

11. Broad Domain
We do NOT attempt to anticipate all We do NOT try to build a formal
questions and build databases. model of the world
Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text.
Structured sources (DBs and KBs) provide background knowledge for interpreting the text.
© 2011 IBM Corporation

12. Hypothesis Scoring
Category: MICHIGAN MANIA
Clue: In 1894 C.W. Post created his warm cereal drink Postum in this
Michigan city Tycor
Answer Scorers can be applied depending on different relations or constraints detected in the Temporal
question. For example, this question focus with modifiers is “Michigan city.” Watson can
detect this as a geospatial relation that indicates the correct answer must be a city spatially
Spatial
located within the sate of Michigan. Popularity
…
Candidate Answers Evidence Feature Scores (Answer Scoring + Passage Scoring)
Doc Rank Pass Rank Ty Cor Geo
General Foods 0 1 0.1 0
Post Foods 2 1 0.1 0
Battle Creek 1 2 0.8 1
Will Keith Kellogg 3 0.1 0
Grand Rapids 0.9 1
1895 0 0.0 0
© 2011 IBM Corporation

13. Passage Scoring
Category: MICHIGAN MANIA
Clue: In 1894 C.W. Post created his warm cereal drink Postum in this
Michigan city
In Deep Evidence Scoring, Watson retrieves evidence for each candidate answer, then evaluates the evidence using a
large number of deep evidence scoring analytics. The evidence for a candidate answer may come from the original
document or passage where the candidate answer was generated, or it may come from an evidence retrieval search
performed by taking the keyword search query from Step 2, replacing the focus terms with the candidate answer, and
retrieving the relevant passages that are found. The passages, or “context” in which the candidate answer occurs are
evaluated as evidence to support or refute the candidate answer as the correct answer for the question.
Battle Creek General Foods
Post Foods
1895: In Battle Creek, Michigan, C.W. 1854 C. W. Post (Charles William) was
Post made the first POSTUM , a cereal
C.W. Post came to the Battle Creek born. He founded the Postum Cereal Co.
beverage. Post created GRAPE-NUTS
sanitarium to cure his upset stomach. in 1895 (renamed General Foods Corp.breakfast
General Foods' products go from
cereal in 1897, and POST TOASTIES
He later created Postum, a cereal- in 1922) to manufacture warm nightcaps (Postum,
(Post's cereals) to Postum cereal
corn flakes in 1908
based coffee substitute Sanka), also wash the pots and pans that its
beverage
The company was incorporated in 1922, foods are cooked in (S.O.S. Scouring Pads
Post Foods, LLC, also known as Post Cereals
having developed from the earlier Postum
(formerly Postum Cereals) was founded by C.W.
Cereal Co. Ltd., founded by C.W. Post
Post. It began in 1895 with the first Postum, a
(1854-1914) in 1895 in Battle Creek, Mich.
"cereal beverage", developed by Post in Battle
After a number of experiments, Post
Creek, Michigan. The first cereal, Grape-Nuts,
marketed his first product-the cereal It was named after C. W. Post, the founder of
was developed in 1897
beverage called Postum-in 1895 the Postum Cereal Company that later
became General Foods. The cereal company
unit was later sold off and is now Post Foods
© 2011 IBM Corporation

14. Merging and Confidence
Category: MICHIGAN MANIA
Clue: In 1894 C.W. Post created his warm cereal drink Postum in this …
In the final processing step, Watson detects variants of the same answer and merges their feature scores together.
Watson then computes the final confidence scores for the candidate answers by applying a series of Machine
Learning models that weight all of the feature scores to produce the final confidence scores.
Candidate Evidence Feature Scores Correct
Answers
Answer
Doc Pass Ty Cor Geo LFACS Term Temp-
Rank Rank Match oral Final Answers Confi-
General Foods 0 1 0.1 0 0.2 22 1 dence
Battle Creek 0.946
Post Foods 2 1 0.1 0 0.4 41 1
Machine
Learning Post Foods 0.152
Battle Creek 1 2 0.8 1 0.5 30 0.9
Model 1895 0.040
Will Keith Kellogg 3 0.1 0 0 23 0.5 Application
Grand Rapids 0.033
Grand Rapids 0.9 1 0 10 0.5
General Foods 0.014
1895 0 0.0 0 0 21 0.6
© 2011 IBM Corporation

15. “Minimal” Deep QA Pipeline
Category: MICHIGAN MANIA
Clue: In 1894 C.W. Post created his warm cereal drink Postum in this
Michigan city
Question
Battle Creek
Final Confidence
Question Primary Hypothesis Hypothesis and
Merging &
Analysis Search Generation Evidence Scoring
Ranking
Document
Search Results
LAT Candidate Evidence Features
R Title Answers
Ty Cor Geo
Final Answers Confi-
Mitchigan 0 General General
Foods dence
City Foods
1 Battle 0.1 0 Battle Creek 0.946
Post
Creek Foods
0.1 0 Post Foods 0.152
2 Post Foods Battle
Creek
0.8 1 1895 0.040
3 Will Keith
Kellogg
© 2011 IBM Corporation

16. Cut to the chase…..
Watson emerges victorious
© 2011 IBM Corporation

17. Technology marches forward…
© 2011 IBM Corporation

18. The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of
over 100 independent algorithms that approximate a Assist & augment human cognition. Watson
solution to the “is this the right answer to the question” depended on primarily a set of background
problem. Achieving winning (human expert) documents (the corpus). The value of having access
performance, required two hallmarks of cognitive to this kind of fact-finding power over a large (and
computing systems: a metric to measure improvements possibly changing) corpus provides a clear
to the system (the winners cloud), and a significant augmentation to human abilities.
ground truth (over 200K Q-A pairs).
Interact in a natural way. Watson was a significant
Speed&Scale. Watson used big data, as well as a
step forward in natural language understanding, the
3000 node cluster for massive computation to get
most basic interface for humans. Say goodbye to
answering speeds down into the 2s range.
your mouse…
© 2011 IBM Corporation

19. The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of
over 100 independent algorithms that approximate a
solution to the “is this the right answer to the question”
problem. Achieving winning (human expert)
100%
performance, required two hallmarks of cognitive
computing systems: a metric to measure improvements
90%
to the system (the winners cloud), and a significant
ground truth (over 200K Q-A pairs).
80%
70%
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
% Answered
© 2011 IBM Corporation

20. The arrival of Cognitive Computing
Symptoms
Assist & augment human cognition. Watson
depended on primarily a set of background
documents (the corpus). The value of having access
Family
History
to this kind of fact-finding power over a large (and
Pa9ent
History
possibly changing) corpus provides a clear
augmentation to human abilities.
Medica9ons
Tests/Findings
Diagnosis
Models
Confidence
Renal failure
Notes/Hypotheses
UTI
Diabetes
Influenza
hypokalemia
Huge
Volumes
of
Texts,
esophogitis
Journals,
References,
DBs
etc.
Most
Confident
Diagnosis:
UTI
© 2011 IBM Corporation

21. The arrival of Cognitive Computing
Speed&Scale. Watson used big data, as well as a
3000 node cluster for massive computation to get
answering speeds down into the 2s range.
© 2011 IBM Corporation

22. The arrival of Cognitive Computing
Interact in a natural way. Watson was a significant
step forward in natural language understanding, the
most basic interface for humans. Say goodbye to
your mouse…
© 2011 IBM Corporation

23. The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of
over 100 independent algorithms that approximate a Assist & augment human cognition. Watson
solution to the “is this the right answer to the question” depended on primarily a set of background
problem. Achieving winning (human expert) documents (the corpus). The value of having access
performance, required two hallmarks of cognitive to this kind of fact-finding power over a large (and
computing systems: a metric to measure improvements possibly changing) corpus provides a clear
to the system (the winners cloud), and a significant augmentation to human abilities.
ground truth (over 200K Q-A pairs).
Interact in a natural way. Watson was a significant
Speed&Scale. Watson used big data, as well as a
step forward in natural language understanding, the
3000 node cluster for massive computation to get
most basic interface for humans. Say goodbye to
answering speeds down into the 2s range.
your mouse…
© 2011 IBM Corporation

24. …and for Social Web
§ First and foremost, social web analytics (e.g. recommendations) and Social
Computing in general lie clearly in the realm of Cognitive Computing
– Uncertainty, natural language, human intelligence
– Inexact solutions that can improve with time, training
– Problems & solutions need metrics to be solvable
§ All cognitive computing systems require ground truth data
– This data is expensive to collect
– Crowdsourcing is a key new technology/approach
§ The user interface moving closer to people
– Natural language, speech, gestures
– In addition, integrating the collection of training data seamlessly into the interface
is a key development
§ Cognitive computing systems require integration of multiple, disparate, data
sources
– Structured, unstructured, semi-structured
– curated, crowdsourced
© 2011 IBM Corporation