SlideShare a Scribd company logo
1 of 82
Download to read offline
Web & Media Group
http://lora-aroyo.org @laroyo
Truth is a Lie
Rules & Semantics from Crowd Perspectives
Web & Media Group
http://lora-aroyo.org @laroyo
“in embracing the diversity of human
beings, we will find a surer way to true
happiness”
Malcolm Gladwell, 2006
in his TED talk “Choice, happiness and spaghetti sauce”
Web & Media Group
http://lora-aroyo.org @laroyo
“If we treat human brains as
processors in a distributed system,
each can perform a small part of a
massive computation”
Luis von Ahn, 2006
“Games with a Purpose”, Computer Magazine
Web & Media Group
http://lora-aroyo.org @laroyo
“The challenge for Watson was to answer questions
posed in every nuance of natural language. The
first step in accomplishing this was to have a lot of
questions and their correct answers. Any cognitive
system must take this first step.”
Chris Welty, 2014
“Cognitive Computing and the Future”, 1st Cognitive Computing Forum
Web & Media Group
http://lora-aroyo.org @laroyo
Take Home Message
True	
  and	
  False	
  is	
  not	
  enough,	
  	
  
There	
  is	
  diversity	
  in	
  human	
  interpreta3on	
  
	
  
CrowdTruth	
  introduces	
  a	
  spa3al	
  representa3on	
  of	
  
meaning	
  that	
  through	
  crowdsourcing	
  	
  
harnesses	
  disagreement	
  
	
  
Using	
  CrowdTruth	
  
untrained	
  workers	
  can	
  be	
  just	
  as	
  reliable	
  as	
  	
  
highly	
  trained	
  experts	
  
Web & Media Group
http://lora-aroyo.org @laroyo
Is this song ….
Passionate
Rousing
Confident
Boisterous
Rowdy
Literate
Poignant
Wistful
Bittersweet
Autumnal
Brooding
Rollicking
Cheerful
Fun
Sweet
Amiable
Good-natured
Humorous
Silly
Campy
Whimsical
Witty
Wry
Aggressive
Fiery
Tense
Anxious
Intense
Volatile
Web & Media Group
http://lora-aroyo.org @laroyo
?Passionate
Rousing
Confident
Boisterous
Rowdy
Literate
Poignant
Wistful
Bittersweet
Autumnal
Brooding
Rollicking
Cheerful
Fun
Sweet
Amiable
Good-natured
Humorous
Silly
Campy
Whimsical
Witty
Wry
Aggressive
Fiery
Tense
Anxious
Intense
Volatile
Is this song ….
Web & Media Group
http://lora-aroyo.org @laroyo
?- category(“That’s not my name”, x)
?Passionate
Rousing
Confident
Boisterous
Rowdy
Literate
Poignant
Wistful
Bittersweet
Autumnal
Brooding
Rollicking
Cheerful
Fun
Sweet
Amiable
Good-natured
Humorous
Silly
Campy
Whimsical
Witty
Wry
Aggressive
Fiery
Tense
Anxious
Intense
Volatile
(Lee and Hu 2012)
Web & Media Group
http://lora-aroyo.org @laroyo
Patients with TYPHUS who were given
ANTIBIOTICS exhibited several side-effects.
With ANTIBIOTICS in short supply,
DDT was used during World War II to
control the insect vectors of TYPHUS.
ANTIBIOTICS are the first line treatment for
indications of TYPHUS.
Does this sentence express
TREATS(Antibiotics, Typhus)?
Web & Media Group
http://lora-aroyo.org @laroyo
TREATS(Antibiotics, Typhus)
Is this True?
Web & Media Group
http://lora-aroyo.org @laroyo
What is the treatment for typhus?
Antibiotic therapy is recommended for both endemic and epidemic
typhus infections because early treatment with antibiotics (for
example, azithromycin, doxycycline, tetracycline, or
chloramphenicol) can cure most people infected with the bacteria.
Consultation with an infectious-disease expert is advised especially
if epidemic typhus or typhus in pregnant females is diagnosed.
Delays in treatment may allow renal, lung, or nervous system
problems to develop. Some patients, especially the elderly, may die.
TREATS(Antibiotics, Typhus)
Is this True?
Web & Media Group
http://lora-aroyo.org @laroyo
TREATS(Corticosteroids, Lupus)
Is this True?
What is the treatment for lupus?
In more severe cases of Lupus, medications that modulate the
immune system (mainly corticosteroids, immunosuppressants) are
used to control the disease and prevent recurrence of symptoms
(known as flares). Depending on the dosage, people who require
steroids may develop Cushing's syndrome. This may subside if and
when the large initial dosage is reduced, but long-term use of even
low doses can cause elevated blood pressure and cataracts.
Numerous new immunosuppressive drugs are being actively tested.
Rather than suppressing the immune system nonspecifically, as
corticosteroids do, they target the responses of individual [types of]
immune cells.
Web & Media Group
http://lora-aroyo.org @laroyo
Human Interpretation
“I will never understand people” “They are the worst”
Web & Media Group
http://lora-aroyo.org @laroyo
human subjectivity, ambiguity & uncertainty of
expression are part of human semantics
Web & Media Group
http://lora-aroyo.org @laroyo
current practices in
computer science (AI)
are based on
fallacies
Web & Media Group
http://lora-aroyo.org @laroyo
single
truth
Web & Media Group
http://lora-aroyo.org @laroyo
experts
rule
Web & Media Group
http://lora-aroyo.org @laroyo
disagreement
is bad
Web & Media Group
http://lora-aroyo.org @laroyo
simplification of context
this all results in
Web & Media Group
http://lora-aroyo.org @laroyo
Web & Media Group
http://lora-aroyo.org @laroyo
this means our systems
understand only
simplified
black & white
world
problem ..
Web & Media Group
http://lora-aroyo.org @laroyo
reality is
complex
diverse
but …
Web & Media Group
http://lora-aroyo.org @laroyo
current
gold standard practices
are not adequate any more
so...
Web & Media Group
http://lora-aroyo.org @laroyo
there is a gap
we need to bridge
and …
Web & Media Group
http://lora-aroyo.org @laroyo
interestingly …
Web & Media Group
http://lora-aroyo.org @laroyo
in humanities …
diversity is well accepted
Web & Media Group
http://lora-aroyo.org @laroyo
in sciences …
diversity is taboo
Web & Media Group
http://lora-aroyo.org @laroyo
despite numerous examples …
string theory debate
Web & Media Group
http://lora-aroyo.org @laroyo
global warming debate
and …
Web & Media Group
http://lora-aroyo.org @laroyo
evolutionary bio. debate
gradualismpunctuated equilibrium
and …
Web & Media Group
http://lora-aroyo.org @laroyo
What can we do?
so …
Web & Media Group
http://lora-aroyo.org @laroyo
•  collective decisions of large groups
of people
•  a group of error-prone decision-
makers can be surprisingly good at
picking the best choice
•  when thumbs up or thumbs down - the
chance of picking the right answer
needs to be > 50%
•  the odds that a most of them will pick
the right answer is greater than any of
them will pick it on their own
•  performance gets better as size grows
1785
Marquis de Condorcet
“wisdom of crowds”
Web & Media Group
http://lora-aroyo.org @laroyo
• asked 787 people to
guess the weight of
an ox
• none got the right
answer
• their collective
guess was almost
perfect
1906
Sir Francis Galton
“wisdom of crowds”
Web & Media Group
http://lora-aroyo.org @laroyo
WWII Math Rosies
1942: Ballistics calculations and flight trajectories
Web & Media Group
http://lora-aroyo.org @laroyo
NASA’s Computer Room
transcribe raw flight data from celluloid film & oscillograph paper
Web & Media Group
http://lora-aroyo.org @laroyo
“Who Wants to Be a Millionaire?”
•  help from individual expert or
audience poll
•  majority of the audience right
91% of the time
•  individuals right only 65% of
the time
“wisdom of crowds”
2004
James Surowiecki
Web & Media Group
http://lora-aroyo.org @laroyo
Crowdsourcing
Human Computing
Web & Media Group
http://lora-aroyo.org @laroyo
the wise crowd
diversity of opinion
independent perspectives
multitude of contexts
gives the big picture
James Surowiecki
Web & Media Group
http://lora-aroyo.org @laroyo
current practices in
computer science (AI)
are based on fallacies
Web & Media Group
http://lora-aroyo.org @laroyo
today’s real-world
complex problem solving
needs understanding of
multiple
perspectives
current practices in
computer science (AI)
are based on fallacies
Web & Media Group
http://lora-aroyo.org @laroyo
single
truth
Web & Media Group
http://lora-aroyo.org @laroyo
multiple
truths
single
truth
Web & Media Group
http://lora-aroyo.org @laroyo
experts
rule
Web & Media Group
http://lora-aroyo.org @laroyo
wisdom
of crowds
experts
rule
Web & Media Group
http://lora-aroyo.org @laroyo
disagreement
is bad
Web & Media Group
http://lora-aroyo.org @laroyo
disagreement
is bad
disagreement
as signal
Web & Media Group
http://lora-aroyo.org @laroyo
“best collective decisions are
result of disagreement,
not consensus or compromise”
James Surowiecki
because …
Web & Media Group
http://lora-aroyo.org @laroyo
can we harness it?
Web & Media Group
http://lora-aroyo.org @laroyo
Is this song ….
?
Passionate
Rousing
Confident
Boisterous
Rowdy
Literate
Poignant
Wistful
Bittersweet
Autumnal
Brooding
Rollicking
Cheerful
Fun
Sweet
Amiable
Good-natured
Humorous
Silly
Campy
Whimsical
Witty
Wry
Aggressive
Fiery
Tense
Anxious
Intense
Volatile
Web & Media Group
http://lora-aroyo.org @laroyo
If “One Truth” & “No Disagreement”
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5
W1 1
W2 1
W3 1
W4 1
W5 1
W6 1
W7
W8
W9 1
W10 1
Totals 1 3 1 2 1
Web & Media Group
http://lora-aroyo.org @laroyo
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W1 1 1 1
W2 1 1 1
W3 1 1 1
W4 1 1
W5 1 1
W6 1 1 1
W7 1 1 1
W8 1 1 1
W9 1 1
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
If “Many Truths” & “Disagreement”
Web & Media Group
http://lora-aroyo.org @laroyo
can indicate
alternative interpretations
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
Disagreement as Signal
can indicate
ambiguity in the
categorisation
can indicate
low quality workers
Web & Media Group
http://lora-aroyo.org @laroyo
Does this sentence express
TREATS(Antibiotics, Typhus)?
Patients with TYPHUS who were given
ANTIBIOTICS exhibited several side-effects.
With ANTIBIOTICS in short supply, DDT was
used during World War II to control the insect
vectors of TYPHUS.
ANTIBIOTICS are the first line treatment for
indications of TYPHUS. 95%
75%
50%
Disagreement is signal here too
Web & Media Group
http://lora-aroyo.org @laroyo
Sentence Treat Prevent Cause Side Effect Diagnose
S1 2
S2 2
S3 2
S4 2
S5 1 1
S6 2
S7 2
S8 1 1
S9 2
S10 2
If “One Truth”, “No Disagreement”
& “Experts”
Web & Media Group
http://lora-aroyo.org @laroyo
•  we rejected 3 fallacies: one truth, experts rule
and disagreement is bad
•  by encouraging disagreement
• we exposed a richer set of possibilities
• that help in identifying, processing &
understanding context
so …
Web & Media Group
http://lora-aroyo.org @laroyo
•  One truth: data collection efforts assume
one correct interpretation for every example
•  All examples are created equal: ground
truth treats all examples the same – either
match the correct result or not
•  Detailed guidelines help: if examples
cause disagreement - add instructions to
limit interpretations
•  Disagreement is bad: increase quality of
annotation data by reducing disagreement
among the annotators
•  One is enough: most of the annotated
examples are evaluated by one person
•  Experts are better: annotators with domain
knowledge provide better annotations
•  Once done, forever valid: annotations are
not updated; new data not aligned with old
7 Myths
myths directly influence the
practice of collecting human
annotated data; Need to be
revised with a new theory of
truth (CrowdTruth)
Lora Aroyo, Chris Welty: Truth is a Lie: 7 Myths about Human Annotation,
AI Magazine 2014.
Web & Media Group
http://lora-aroyo.org @laroyo
Web & Media Group
http://lora-aroyo.org @laroyo
crowdtruth.org
Web & Media Group
http://lora-aroyo.org @laroyo
crowdtruth.org
a new approach to understanding semantics
harnessing the power & diversity of the crowd
Web & Media Group
http://lora-aroyo.org @laroyo
crowdtruth.org
informs a vector space model of truth
instead of a boolean, fuzzy or statistical model
Web & Media Group
http://lora-aroyo.org @laroyo
observerreferent
sign
has	
  
triangle of
reference
(Ogden & Richards, 1923)
Web & Media Group
http://lora-aroyo.org @laroyo
crowd
annotatorannotation
example
annotation	
  
choices	
  
Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014.
triangle of
reference
Web & Media Group
http://lora-aroyo.org @laroyo
crowd
annotatorannotation
example
annotation	
  
choices	
  
passionate,	
   rollicking,	
   literate,	
   humorous,	
  silly,	
   aggressive,	
  fiery,	
   does	
  not	
  fit	
  into	
  
rousing,	
   cheerful,	
  fun,	
   poignant,	
  wis9ul,	
   campy,	
  quirky,	
   tense,	
  anxious,	
   any	
  of	
  the	
  5	
  
confident,	
   sweet,	
  amiable,	
   bi>ersweet,	
   whimsical,	
  wi>y,	
   intense,	
  vola?le,	
   clusters	
  
boisterous,	
   good-­‐natured	
   autumnal,	
   wry	
   visceral	
   	
  	
  
rowdy	
   	
  	
   brooding	
   	
  	
   	
  	
   	
  	
  
Category 1
Category 3
Category 5
Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014.
triangle of
reference
Web & Media Group
http://lora-aroyo.org @laroyo
crowd
annotatorannotation
example
annotation	
  
choices	
  
treat	
   prevent	
   side	
  effect	
   diagnose	
   cause	
   other	
  
Treat
Cause
Prevent
Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014.
ANTIBIOTICS are the first line
treatment for indications of TYPHUS.
triangle of
reference
Web & Media Group
http://lora-aroyo.org @laroyo
1 1 1
Worker Vector
Web & Media Group
http://lora-aroyo.org @laroyo
1 1 1
1 1
1
1 1
1 1
1 1
1
1
1
0 1 1 0 0 4 3 0 0 5 1 0
Sentence Vector
OUR
NEW THEORY OF
TRUTH
Web & Media Group
http://lora-aroyo.org @laroyo
Unclear relationship between the two arguments
reflected in the disagreement
Sentence Clarity
Feeling the way the CHEST expands (PALPATION), can identify areas of
the lung that are full of fluid.
?PALPATIONIs CHEST related to
diagnose location associated
with
is_a otherpart_of
0 0 02 3 0 0 0 1 0 0 44 1
Web & Media Group
http://lora-aroyo.org @laroyo
Clearly expressed relation between the two
arguments reflected in the agreement
Sentence Clarity
?CONJUNCTIVITISHYPERAEMIA related toIs
0 0 0 1 0 0 0 013 0 0 0 0 0
symptomcause
Redness (HYPERAEMIA), irritation (chemosis) and watering (epiphora)
of the eyes are symptoms common to all forms of CONJUNCTIVITIS.
Web & Media Group
http://lora-aroyo.org @laroyo
1 13
4 3 1
7 2 1
6 2 1 1
1 1 1 1
3 3 1 1
8 3 1
5 7 1
1 2 1
Relation Similarity
Web & Media Group
http://lora-aroyo.org @laroyo
Measures how clearly a sentence expresses a relation
0 1 1 0 0 4 3 0 0 5 1 0
Unit vector
for relation R6
Sentence
Vector
Cosine = .55 confidence
Sentence-Relation Score
Web & Media Group
http://lora-aroyo.org @laroyo
Measured per worker averaged over all sentences
(whether worker consistently disagrees with everyone else)
Worker-Sentence Disagreement
0 1 1 0 0 4 3 0 0 5 1 0
Worker’s
sentence vector
Sentence
Vector
AVG (Cosine)
Worker Quality
Web & Media Group
http://lora-aroyo.org @laroyo
Measured for pairs of workers / sentence
averaged over all sentences
(identify communities of though)
Worker-Worker Disagreement
0 1 1 0 0 1 0 0 0 0 1 0
Worker 1
sentence vector
Worker 2
Vector
AVG (Cosine)
Worker Similarity
Web & Media Group
http://lora-aroyo.org @laroyo
Web & Media Group
http://lora-aroyo.org @laroyo
Example Results
Web & Media Group
http://lora-aroyo.org @laroyo
Example Results
Web & Media Group
http://lora-aroyo.org @laroyo
Example Results
Web & Media Group
http://lora-aroyo.org @laroyo
crowdtruth.org
experimental results show that
CrowdTruth is a better way of capturing context
a more accurate way to predict & explain truth
Web & Media Group
http://lora-aroyo.org @laroyo
Training a Relation Extraction Classifier
F1
Cost per
sentence
CrowdTruth 0.642 $0.66
Expert Annotator 0.638 $2.00
Single Annotator 0.492 $0.08
“wisdom of the crowd”
provides training data that is at least as good
if not better than experts
only with proper analytic framework for
harnessing disagreement from the crowd
Web & Media Group
http://lora-aroyo.org @laroyo
Learning Curves
(crowd with pos./neg. threshold at 0.5
above 400 sent.: crowd consistently over baseline & single
above 600 sent.: crowd out-performs experts
Web & Media Group
http://lora-aroyo.org @laroyo
Learning Curves
Web & Media Group
http://lora-aroyo.org @laroyo
Take Home Message
True	
  and	
  False	
  is	
  not	
  enough,	
  	
  
There	
  is	
  diversity	
  in	
  human	
  interpreta3on	
  
	
  
CrowdTruth	
  introduces	
  a	
  spa3al	
  representa3on	
  of	
  
meaning	
  that	
  through	
  crowdsourcing	
  	
  
harnesses	
  disagreement	
  
	
  
Using	
  CrowdTruth	
  
untrained	
  workers	
  can	
  be	
  just	
  as	
  reliable	
  as	
  	
  
highly	
  trained	
  experts	
  
Web & Media Group
http://lora-aroyo.org @laroyo
https://www.youtube.com/watch?v=CyAI_lVUdzM
To be AND not to be: quantum
intelligence?
Lora Aroyo & Chris Welty

More Related Content

What's hot

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Lora Aroyo
 
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Lora Aroyo
 
Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Lora Aroyo
 
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Lora Aroyo
 
Social Web 2014: Final Presentations (Part I)
Social Web 2014: Final Presentations (Part I)Social Web 2014: Final Presentations (Part I)
Social Web 2014: Final Presentations (Part I)Lora Aroyo
 
ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016oanainel
 
"What is Data Science?"
"What is Data Science?""What is Data Science?"
"What is Data Science?"Renee Teate
 
State of RecSys: Recap of RecSys 2012
State of RecSys: Recap of RecSys 2012State of RecSys: Recap of RecSys 2012
State of RecSys: Recap of RecSys 2012Alan Said
 
Finding the Phoenix: Feathers, Flight & the Future of Libraries
Finding the Phoenix: Feathers, Flight & the Future of LibrariesFinding the Phoenix: Feathers, Flight & the Future of Libraries
Finding the Phoenix: Feathers, Flight & the Future of Librarieshblowers
 

What's hot (10)

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
 
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
 
Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)
 
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
 
Social Web 2014: Final Presentations (Part I)
Social Web 2014: Final Presentations (Part I)Social Web 2014: Final Presentations (Part I)
Social Web 2014: Final Presentations (Part I)
 
ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016
 
"What is Data Science?"
"What is Data Science?""What is Data Science?"
"What is Data Science?"
 
Living Online
Living OnlineLiving Online
Living Online
 
State of RecSys: Recap of RecSys 2012
State of RecSys: Recap of RecSys 2012State of RecSys: Recap of RecSys 2012
State of RecSys: Recap of RecSys 2012
 
Finding the Phoenix: Feathers, Flight & the Future of Libraries
Finding the Phoenix: Feathers, Flight & the Future of LibrariesFinding the Phoenix: Feathers, Flight & the Future of Libraries
Finding the Phoenix: Feathers, Flight & the Future of Libraries
 

Viewers also liked

(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 Lora Aroyo
 
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Lora Aroyo
 
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataVisualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataCrowdTruth
 
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?CrowdTruth
 
Crowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain QuestionsCrowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain QuestionsBenjamin Timmermans
 
Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...Benjamin Timmermans
 
Towards Better Media Understanding and Searchability
Towards Better Media Understanding and SearchabilityTowards Better Media Understanding and Searchability
Towards Better Media Understanding and Searchabilityoanainel
 
Dive+@ICTOpen2017
Dive+@ICTOpen2017Dive+@ICTOpen2017
Dive+@ICTOpen2017oanainel
 
Dive+ NL eScience symposium 2015
Dive+ NL eScience symposium 2015Dive+ NL eScience symposium 2015
Dive+ NL eScience symposium 2015CrowdTruth
 
CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015Lora Aroyo
 
Boosting Named Entity Extraction through Crowdsourcing
Boosting Named Entity Extraction through CrowdsourcingBoosting Named Entity Extraction through Crowdsourcing
Boosting Named Entity Extraction through Crowdsourcingoanainel
 
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital AgeEuropeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital AgeLora Aroyo
 
DIVE Semantic Web Challenge Presentation
DIVE Semantic Web Challenge Presentation DIVE Semantic Web Challenge Presentation
DIVE Semantic Web Challenge Presentation Victor de Boer
 
Genuine semantic publishing
Genuine semantic publishingGenuine semantic publishing
Genuine semantic publishingTobias Kuhn
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumLora Aroyo
 
CrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsCrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsLora Aroyo
 
Closing Event - Watson Innovation Course
Closing Event - Watson Innovation CourseClosing Event - Watson Innovation Course
Closing Event - Watson Innovation CourseLora Aroyo
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+Lora Aroyo
 
Keynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventKeynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventLora Aroyo
 

Viewers also liked (20)

(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
 
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
 
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataVisualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
 
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
 
Crowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain QuestionsCrowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain Questions
 
Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...
 
Towards Better Media Understanding and Searchability
Towards Better Media Understanding and SearchabilityTowards Better Media Understanding and Searchability
Towards Better Media Understanding and Searchability
 
Dive+@ICTOpen2017
Dive+@ICTOpen2017Dive+@ICTOpen2017
Dive+@ICTOpen2017
 
Dive+ NL eScience symposium 2015
Dive+ NL eScience symposium 2015Dive+ NL eScience symposium 2015
Dive+ NL eScience symposium 2015
 
CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015
 
Boosting Named Entity Extraction through Crowdsourcing
Boosting Named Entity Extraction through CrowdsourcingBoosting Named Entity Extraction through Crowdsourcing
Boosting Named Entity Extraction through Crowdsourcing
 
Kick-off meeting Linkflows project
Kick-off meeting Linkflows projectKick-off meeting Linkflows project
Kick-off meeting Linkflows project
 
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital AgeEuropeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
 
DIVE Semantic Web Challenge Presentation
DIVE Semantic Web Challenge Presentation DIVE Semantic Web Challenge Presentation
DIVE Semantic Web Challenge Presentation
 
Genuine semantic publishing
Genuine semantic publishingGenuine semantic publishing
Genuine semantic publishing
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
 
CrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsCrowdTruth for Digital Hermeneutics
CrowdTruth for Digital Hermeneutics
 
Closing Event - Watson Innovation Course
Closing Event - Watson Innovation CourseClosing Event - Watson Innovation Course
Closing Event - Watson Innovation Course
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
 
Keynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventKeynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program Event
 

Similar to Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)

Using cognitive tools in robots dealing with people with dementia
Using cognitive tools in robots dealing with people with dementiaUsing cognitive tools in robots dealing with people with dementia
Using cognitive tools in robots dealing with people with dementiaValentina Presutti
 
Current and Future Trends in e-learning - MEd in Surgical Education - Imperia...
Current and Future Trends in e-learning - MEd in Surgical Education - Imperia...Current and Future Trends in e-learning - MEd in Surgical Education - Imperia...
Current and Future Trends in e-learning - MEd in Surgical Education - Imperia...mtorotro
 
Project Management Infomaton System
Project Management Infomaton SystemProject Management Infomaton System
Project Management Infomaton SystemJennifer Robinson
 
Peter nguyen- Capture and influence the consumers opinions in pharmaceutical ...
Peter nguyen- Capture and influence the consumers opinions in pharmaceutical ...Peter nguyen- Capture and influence the consumers opinions in pharmaceutical ...
Peter nguyen- Capture and influence the consumers opinions in pharmaceutical ...Nguyen Hien
 
Using Smart Technology to Improve Global Health Initiatives
Using Smart Technology to Improve Global Health InitiativesUsing Smart Technology to Improve Global Health Initiatives
Using Smart Technology to Improve Global Health Initiativesguestfddc18
 
Secure Data Transmission Through Multiple Hops
Secure Data Transmission Through Multiple HopsSecure Data Transmission Through Multiple Hops
Secure Data Transmission Through Multiple HopsTara Smith
 
(New) Media Education: Teachers, join the conversation
(New) Media Education: Teachers, join the conversation (New) Media Education: Teachers, join the conversation
(New) Media Education: Teachers, join the conversation Dino Amenduni
 
Social Media Advantages And Disadvantages
Social Media Advantages And DisadvantagesSocial Media Advantages And Disadvantages
Social Media Advantages And DisadvantagesAmy Washington
 
Engaging with Patients Online: The do’s and don’t’s, and what’s to gain
Engaging with Patients Online: The do’s and don’t’s, and what’s to gainEngaging with Patients Online: The do’s and don’t’s, and what’s to gain
Engaging with Patients Online: The do’s and don’t’s, and what’s to gainKatja Reuter, PhD
 
The Use of Social Media in Strata Complexes
The Use of Social Media in Strata ComplexesThe Use of Social Media in Strata Complexes
The Use of Social Media in Strata ComplexesTEYS Lawyers
 
Social Mediafor Development Educators Belfastpres
Social Mediafor Development Educators BelfastpresSocial Mediafor Development Educators Belfastpres
Social Mediafor Development Educators BelfastpresEimear IDEAIreland
 
Social Mediafor Development Educators Belfastpres
Social Mediafor Development Educators BelfastpresSocial Mediafor Development Educators Belfastpres
Social Mediafor Development Educators BelfastpresEimear IDEAIreland
 
The renaissance brand
The renaissance brandThe renaissance brand
The renaissance brandYoni Kish
 
Michael Edson @ Forum One: Strategy and Audience (long version)
Michael Edson @ Forum One: Strategy and Audience (long version)Michael Edson @ Forum One: Strategy and Audience (long version)
Michael Edson @ Forum One: Strategy and Audience (long version)Michael Edson
 

Similar to Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote) (20)

Using cognitive tools in robots dealing with people with dementia
Using cognitive tools in robots dealing with people with dementiaUsing cognitive tools in robots dealing with people with dementia
Using cognitive tools in robots dealing with people with dementia
 
Current and Future Trends in e-learning - MEd in Surgical Education - Imperia...
Current and Future Trends in e-learning - MEd in Surgical Education - Imperia...Current and Future Trends in e-learning - MEd in Surgical Education - Imperia...
Current and Future Trends in e-learning - MEd in Surgical Education - Imperia...
 
Project Management Infomaton System
Project Management Infomaton SystemProject Management Infomaton System
Project Management Infomaton System
 
Peter nguyen- Capture and influence the consumers opinions in pharmaceutical ...
Peter nguyen- Capture and influence the consumers opinions in pharmaceutical ...Peter nguyen- Capture and influence the consumers opinions in pharmaceutical ...
Peter nguyen- Capture and influence the consumers opinions in pharmaceutical ...
 
FOAMed: An introduction
FOAMed: An introductionFOAMed: An introduction
FOAMed: An introduction
 
Using Smart Technology to Improve Global Health Initiatives
Using Smart Technology to Improve Global Health InitiativesUsing Smart Technology to Improve Global Health Initiatives
Using Smart Technology to Improve Global Health Initiatives
 
Secure Data Transmission Through Multiple Hops
Secure Data Transmission Through Multiple HopsSecure Data Transmission Through Multiple Hops
Secure Data Transmission Through Multiple Hops
 
Social Technologies for eHealth
Social Technologies for eHealthSocial Technologies for eHealth
Social Technologies for eHealth
 
Health Care Collaboration & Community in Virtual Worlds & Second Life
Health Care Collaboration & Community in Virtual Worlds & Second LifeHealth Care Collaboration & Community in Virtual Worlds & Second Life
Health Care Collaboration & Community in Virtual Worlds & Second Life
 
Engaging participation from the chemistry community
Engaging participation from the chemistry communityEngaging participation from the chemistry community
Engaging participation from the chemistry community
 
Science 2.0
Science 2.0Science 2.0
Science 2.0
 
(New) Media Education: Teachers, join the conversation
(New) Media Education: Teachers, join the conversation (New) Media Education: Teachers, join the conversation
(New) Media Education: Teachers, join the conversation
 
Social Technologies for Informaticians and Researchers
Social Technologies for Informaticians and ResearchersSocial Technologies for Informaticians and Researchers
Social Technologies for Informaticians and Researchers
 
Social Media Advantages And Disadvantages
Social Media Advantages And DisadvantagesSocial Media Advantages And Disadvantages
Social Media Advantages And Disadvantages
 
Engaging with Patients Online: The do’s and don’t’s, and what’s to gain
Engaging with Patients Online: The do’s and don’t’s, and what’s to gainEngaging with Patients Online: The do’s and don’t’s, and what’s to gain
Engaging with Patients Online: The do’s and don’t’s, and what’s to gain
 
The Use of Social Media in Strata Complexes
The Use of Social Media in Strata ComplexesThe Use of Social Media in Strata Complexes
The Use of Social Media in Strata Complexes
 
Social Mediafor Development Educators Belfastpres
Social Mediafor Development Educators BelfastpresSocial Mediafor Development Educators Belfastpres
Social Mediafor Development Educators Belfastpres
 
Social Mediafor Development Educators Belfastpres
Social Mediafor Development Educators BelfastpresSocial Mediafor Development Educators Belfastpres
Social Mediafor Development Educators Belfastpres
 
The renaissance brand
The renaissance brandThe renaissance brand
The renaissance brand
 
Michael Edson @ Forum One: Strategy and Audience (long version)
Michael Edson @ Forum One: Strategy and Audience (long version)Michael Edson @ Forum One: Strategy and Audience (long version)
Michael Edson @ Forum One: Strategy and Audience (long version)
 

More from Lora Aroyo

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfLora Aroyo
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningLora Aroyo
 
Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AILora Aroyo
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumLora Aroyo
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorLora Aroyo
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataLora Aroyo
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumLora Aroyo
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18Lora Aroyo
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsLora Aroyo
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesLora Aroyo
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...Lora Aroyo
 
"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to SnapchatLora Aroyo
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyLora Aroyo
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...Lora Aroyo
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Lora Aroyo
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordLora Aroyo
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...Lora Aroyo
 

More from Lora Aroyo (17)

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
 
Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AI
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH Symposium
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP Demonstrator
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked Data
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithms
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & Machines
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
 
"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening Ceremony
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @Oxford
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
 

Recently uploaded

Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 

Recently uploaded (20)

Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 

Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)

  • 1. Web & Media Group http://lora-aroyo.org @laroyo Truth is a Lie Rules & Semantics from Crowd Perspectives
  • 2. Web & Media Group http://lora-aroyo.org @laroyo “in embracing the diversity of human beings, we will find a surer way to true happiness” Malcolm Gladwell, 2006 in his TED talk “Choice, happiness and spaghetti sauce”
  • 3. Web & Media Group http://lora-aroyo.org @laroyo “If we treat human brains as processors in a distributed system, each can perform a small part of a massive computation” Luis von Ahn, 2006 “Games with a Purpose”, Computer Magazine
  • 4. Web & Media Group http://lora-aroyo.org @laroyo “The challenge for Watson was to answer questions posed in every nuance of natural language. The first step in accomplishing this was to have a lot of questions and their correct answers. Any cognitive system must take this first step.” Chris Welty, 2014 “Cognitive Computing and the Future”, 1st Cognitive Computing Forum
  • 5. Web & Media Group http://lora-aroyo.org @laroyo Take Home Message True  and  False  is  not  enough,     There  is  diversity  in  human  interpreta3on     CrowdTruth  introduces  a  spa3al  representa3on  of   meaning  that  through  crowdsourcing     harnesses  disagreement     Using  CrowdTruth   untrained  workers  can  be  just  as  reliable  as     highly  trained  experts  
  • 6. Web & Media Group http://lora-aroyo.org @laroyo Is this song …. Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile
  • 7. Web & Media Group http://lora-aroyo.org @laroyo ?Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile Is this song ….
  • 8. Web & Media Group http://lora-aroyo.org @laroyo ?- category(“That’s not my name”, x) ?Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile (Lee and Hu 2012)
  • 9. Web & Media Group http://lora-aroyo.org @laroyo Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects. With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. ANTIBIOTICS are the first line treatment for indications of TYPHUS. Does this sentence express TREATS(Antibiotics, Typhus)?
  • 10. Web & Media Group http://lora-aroyo.org @laroyo TREATS(Antibiotics, Typhus) Is this True?
  • 11. Web & Media Group http://lora-aroyo.org @laroyo What is the treatment for typhus? Antibiotic therapy is recommended for both endemic and epidemic typhus infections because early treatment with antibiotics (for example, azithromycin, doxycycline, tetracycline, or chloramphenicol) can cure most people infected with the bacteria. Consultation with an infectious-disease expert is advised especially if epidemic typhus or typhus in pregnant females is diagnosed. Delays in treatment may allow renal, lung, or nervous system problems to develop. Some patients, especially the elderly, may die. TREATS(Antibiotics, Typhus) Is this True?
  • 12. Web & Media Group http://lora-aroyo.org @laroyo TREATS(Corticosteroids, Lupus) Is this True? What is the treatment for lupus? In more severe cases of Lupus, medications that modulate the immune system (mainly corticosteroids, immunosuppressants) are used to control the disease and prevent recurrence of symptoms (known as flares). Depending on the dosage, people who require steroids may develop Cushing's syndrome. This may subside if and when the large initial dosage is reduced, but long-term use of even low doses can cause elevated blood pressure and cataracts. Numerous new immunosuppressive drugs are being actively tested. Rather than suppressing the immune system nonspecifically, as corticosteroids do, they target the responses of individual [types of] immune cells.
  • 13. Web & Media Group http://lora-aroyo.org @laroyo Human Interpretation “I will never understand people” “They are the worst”
  • 14. Web & Media Group http://lora-aroyo.org @laroyo human subjectivity, ambiguity & uncertainty of expression are part of human semantics
  • 15. Web & Media Group http://lora-aroyo.org @laroyo current practices in computer science (AI) are based on fallacies
  • 16. Web & Media Group http://lora-aroyo.org @laroyo single truth
  • 17. Web & Media Group http://lora-aroyo.org @laroyo experts rule
  • 18. Web & Media Group http://lora-aroyo.org @laroyo disagreement is bad
  • 19. Web & Media Group http://lora-aroyo.org @laroyo simplification of context this all results in
  • 20. Web & Media Group http://lora-aroyo.org @laroyo
  • 21. Web & Media Group http://lora-aroyo.org @laroyo this means our systems understand only simplified black & white world problem ..
  • 22. Web & Media Group http://lora-aroyo.org @laroyo reality is complex diverse but …
  • 23. Web & Media Group http://lora-aroyo.org @laroyo current gold standard practices are not adequate any more so...
  • 24. Web & Media Group http://lora-aroyo.org @laroyo there is a gap we need to bridge and …
  • 25. Web & Media Group http://lora-aroyo.org @laroyo interestingly …
  • 26. Web & Media Group http://lora-aroyo.org @laroyo in humanities … diversity is well accepted
  • 27. Web & Media Group http://lora-aroyo.org @laroyo in sciences … diversity is taboo
  • 28. Web & Media Group http://lora-aroyo.org @laroyo despite numerous examples … string theory debate
  • 29. Web & Media Group http://lora-aroyo.org @laroyo global warming debate and …
  • 30. Web & Media Group http://lora-aroyo.org @laroyo evolutionary bio. debate gradualismpunctuated equilibrium and …
  • 31. Web & Media Group http://lora-aroyo.org @laroyo What can we do? so …
  • 32. Web & Media Group http://lora-aroyo.org @laroyo •  collective decisions of large groups of people •  a group of error-prone decision- makers can be surprisingly good at picking the best choice •  when thumbs up or thumbs down - the chance of picking the right answer needs to be > 50% •  the odds that a most of them will pick the right answer is greater than any of them will pick it on their own •  performance gets better as size grows 1785 Marquis de Condorcet “wisdom of crowds”
  • 33. Web & Media Group http://lora-aroyo.org @laroyo • asked 787 people to guess the weight of an ox • none got the right answer • their collective guess was almost perfect 1906 Sir Francis Galton “wisdom of crowds”
  • 34. Web & Media Group http://lora-aroyo.org @laroyo WWII Math Rosies 1942: Ballistics calculations and flight trajectories
  • 35. Web & Media Group http://lora-aroyo.org @laroyo NASA’s Computer Room transcribe raw flight data from celluloid film & oscillograph paper
  • 36. Web & Media Group http://lora-aroyo.org @laroyo “Who Wants to Be a Millionaire?” •  help from individual expert or audience poll •  majority of the audience right 91% of the time •  individuals right only 65% of the time “wisdom of crowds” 2004 James Surowiecki
  • 37. Web & Media Group http://lora-aroyo.org @laroyo Crowdsourcing Human Computing
  • 38. Web & Media Group http://lora-aroyo.org @laroyo the wise crowd diversity of opinion independent perspectives multitude of contexts gives the big picture James Surowiecki
  • 39. Web & Media Group http://lora-aroyo.org @laroyo current practices in computer science (AI) are based on fallacies
  • 40. Web & Media Group http://lora-aroyo.org @laroyo today’s real-world complex problem solving needs understanding of multiple perspectives current practices in computer science (AI) are based on fallacies
  • 41. Web & Media Group http://lora-aroyo.org @laroyo single truth
  • 42. Web & Media Group http://lora-aroyo.org @laroyo multiple truths single truth
  • 43. Web & Media Group http://lora-aroyo.org @laroyo experts rule
  • 44. Web & Media Group http://lora-aroyo.org @laroyo wisdom of crowds experts rule
  • 45. Web & Media Group http://lora-aroyo.org @laroyo disagreement is bad
  • 46. Web & Media Group http://lora-aroyo.org @laroyo disagreement is bad disagreement as signal
  • 47. Web & Media Group http://lora-aroyo.org @laroyo “best collective decisions are result of disagreement, not consensus or compromise” James Surowiecki because …
  • 48. Web & Media Group http://lora-aroyo.org @laroyo can we harness it?
  • 49. Web & Media Group http://lora-aroyo.org @laroyo Is this song …. ? Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile
  • 50. Web & Media Group http://lora-aroyo.org @laroyo If “One Truth” & “No Disagreement” Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 W1 1 W2 1 W3 1 W4 1 W5 1 W6 1 W7 W8 W9 1 W10 1 Totals 1 3 1 2 1
  • 51. Web & Media Group http://lora-aroyo.org @laroyo Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other W1 1 1 1 W2 1 1 1 W3 1 1 1 W4 1 1 W5 1 1 W6 1 1 1 W7 1 1 1 W8 1 1 1 W9 1 1 W10 1 1 1 1 1 Totals 3 5 6 5 2 8 If “Many Truths” & “Disagreement”
  • 52. Web & Media Group http://lora-aroyo.org @laroyo can indicate alternative interpretations Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other W10 1 1 1 1 1 Totals 3 5 6 5 2 8 Disagreement as Signal can indicate ambiguity in the categorisation can indicate low quality workers
  • 53. Web & Media Group http://lora-aroyo.org @laroyo Does this sentence express TREATS(Antibiotics, Typhus)? Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects. With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. ANTIBIOTICS are the first line treatment for indications of TYPHUS. 95% 75% 50% Disagreement is signal here too
  • 54. Web & Media Group http://lora-aroyo.org @laroyo Sentence Treat Prevent Cause Side Effect Diagnose S1 2 S2 2 S3 2 S4 2 S5 1 1 S6 2 S7 2 S8 1 1 S9 2 S10 2 If “One Truth”, “No Disagreement” & “Experts”
  • 55. Web & Media Group http://lora-aroyo.org @laroyo •  we rejected 3 fallacies: one truth, experts rule and disagreement is bad •  by encouraging disagreement • we exposed a richer set of possibilities • that help in identifying, processing & understanding context so …
  • 56. Web & Media Group http://lora-aroyo.org @laroyo •  One truth: data collection efforts assume one correct interpretation for every example •  All examples are created equal: ground truth treats all examples the same – either match the correct result or not •  Detailed guidelines help: if examples cause disagreement - add instructions to limit interpretations •  Disagreement is bad: increase quality of annotation data by reducing disagreement among the annotators •  One is enough: most of the annotated examples are evaluated by one person •  Experts are better: annotators with domain knowledge provide better annotations •  Once done, forever valid: annotations are not updated; new data not aligned with old 7 Myths myths directly influence the practice of collecting human annotated data; Need to be revised with a new theory of truth (CrowdTruth) Lora Aroyo, Chris Welty: Truth is a Lie: 7 Myths about Human Annotation, AI Magazine 2014.
  • 57. Web & Media Group http://lora-aroyo.org @laroyo
  • 58. Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org
  • 59. Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org a new approach to understanding semantics harnessing the power & diversity of the crowd
  • 60. Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org informs a vector space model of truth instead of a boolean, fuzzy or statistical model
  • 61. Web & Media Group http://lora-aroyo.org @laroyo observerreferent sign has   triangle of reference (Ogden & Richards, 1923)
  • 62. Web & Media Group http://lora-aroyo.org @laroyo crowd annotatorannotation example annotation   choices   Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014. triangle of reference
  • 63. Web & Media Group http://lora-aroyo.org @laroyo crowd annotatorannotation example annotation   choices   passionate,   rollicking,   literate,   humorous,  silly,   aggressive,  fiery,   does  not  fit  into   rousing,   cheerful,  fun,   poignant,  wis9ul,   campy,  quirky,   tense,  anxious,   any  of  the  5   confident,   sweet,  amiable,   bi>ersweet,   whimsical,  wi>y,   intense,  vola?le,   clusters   boisterous,   good-­‐natured   autumnal,   wry   visceral       rowdy       brooding               Category 1 Category 3 Category 5 Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014. triangle of reference
  • 64. Web & Media Group http://lora-aroyo.org @laroyo crowd annotatorannotation example annotation   choices   treat   prevent   side  effect   diagnose   cause   other   Treat Cause Prevent Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014. ANTIBIOTICS are the first line treatment for indications of TYPHUS. triangle of reference
  • 65. Web & Media Group http://lora-aroyo.org @laroyo 1 1 1 Worker Vector
  • 66. Web & Media Group http://lora-aroyo.org @laroyo 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 0 4 3 0 0 5 1 0 Sentence Vector OUR NEW THEORY OF TRUTH
  • 67. Web & Media Group http://lora-aroyo.org @laroyo Unclear relationship between the two arguments reflected in the disagreement Sentence Clarity Feeling the way the CHEST expands (PALPATION), can identify areas of the lung that are full of fluid. ?PALPATIONIs CHEST related to diagnose location associated with is_a otherpart_of 0 0 02 3 0 0 0 1 0 0 44 1
  • 68. Web & Media Group http://lora-aroyo.org @laroyo Clearly expressed relation between the two arguments reflected in the agreement Sentence Clarity ?CONJUNCTIVITISHYPERAEMIA related toIs 0 0 0 1 0 0 0 013 0 0 0 0 0 symptomcause Redness (HYPERAEMIA), irritation (chemosis) and watering (epiphora) of the eyes are symptoms common to all forms of CONJUNCTIVITIS.
  • 69. Web & Media Group http://lora-aroyo.org @laroyo 1 13 4 3 1 7 2 1 6 2 1 1 1 1 1 1 3 3 1 1 8 3 1 5 7 1 1 2 1 Relation Similarity
  • 70. Web & Media Group http://lora-aroyo.org @laroyo Measures how clearly a sentence expresses a relation 0 1 1 0 0 4 3 0 0 5 1 0 Unit vector for relation R6 Sentence Vector Cosine = .55 confidence Sentence-Relation Score
  • 71. Web & Media Group http://lora-aroyo.org @laroyo Measured per worker averaged over all sentences (whether worker consistently disagrees with everyone else) Worker-Sentence Disagreement 0 1 1 0 0 4 3 0 0 5 1 0 Worker’s sentence vector Sentence Vector AVG (Cosine) Worker Quality
  • 72. Web & Media Group http://lora-aroyo.org @laroyo Measured for pairs of workers / sentence averaged over all sentences (identify communities of though) Worker-Worker Disagreement 0 1 1 0 0 1 0 0 0 0 1 0 Worker 1 sentence vector Worker 2 Vector AVG (Cosine) Worker Similarity
  • 73. Web & Media Group http://lora-aroyo.org @laroyo
  • 74. Web & Media Group http://lora-aroyo.org @laroyo Example Results
  • 75. Web & Media Group http://lora-aroyo.org @laroyo Example Results
  • 76. Web & Media Group http://lora-aroyo.org @laroyo Example Results
  • 77. Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org experimental results show that CrowdTruth is a better way of capturing context a more accurate way to predict & explain truth
  • 78. Web & Media Group http://lora-aroyo.org @laroyo Training a Relation Extraction Classifier F1 Cost per sentence CrowdTruth 0.642 $0.66 Expert Annotator 0.638 $2.00 Single Annotator 0.492 $0.08 “wisdom of the crowd” provides training data that is at least as good if not better than experts only with proper analytic framework for harnessing disagreement from the crowd
  • 79. Web & Media Group http://lora-aroyo.org @laroyo Learning Curves (crowd with pos./neg. threshold at 0.5 above 400 sent.: crowd consistently over baseline & single above 600 sent.: crowd out-performs experts
  • 80. Web & Media Group http://lora-aroyo.org @laroyo Learning Curves
  • 81. Web & Media Group http://lora-aroyo.org @laroyo Take Home Message True  and  False  is  not  enough,     There  is  diversity  in  human  interpreta3on     CrowdTruth  introduces  a  spa3al  representa3on  of   meaning  that  through  crowdsourcing     harnesses  disagreement     Using  CrowdTruth   untrained  workers  can  be  just  as  reliable  as     highly  trained  experts  
  • 82. Web & Media Group http://lora-aroyo.org @laroyo https://www.youtube.com/watch?v=CyAI_lVUdzM To be AND not to be: quantum intelligence? Lora Aroyo & Chris Welty