Successfully comprehending stories involves integration of the story information with the reader's own background knowledge. A prerequisite, then, of building automated story understanding systems is the availability of such background knowledge. We take the approach that knowledge appropriate for story understanding can be gathered by sourcing the task to the crowd. Our methodology centers on breaking this task into a sequence of more specific tasks, so that human participants not only identify relevant knowledge, but also convert it into a machine-readable form, generalize it, and evaluate its appropriateness. These individual tasks are presented to human participants as missions in an online game, offering them, in this manner, an incentive for their participation. We report on an initial deployment of the game, and discuss our ongoing work for integrating the knowledge gathering task into a full-fledged story understanding engine.

1. Gathering Background Knowledge for
Story Understanding through
Crowdsourcing
Christos T. Rodosthenous & Loizos Michael
Computational Cognition Lab, Open University of Cyprus

2. Introduction
General research problem
Acquisition of Background Knowledge to be used
by an automated story understanding system
Approach to the problem
Develop a method / system to facilitate
knowledge acquisition using crowdsourcing
techniques
Source task to the crowd

3. Engine Architecture

4. Knowledge Representation
Gather knowledge in a machine-readable form
High-level version of the Event Calculus (Michael,
2010)
A fluent F: is an object whose value can change
through the course of time
An action A: event that occurs at a specific time-
point
A literal L: fluent or an action, or their negation

5. Knowledge Representation
Φ implies L: constraints that hold at each story time-
point
e.g., person(X) implies can(X,think)
Φ causes L: capture the conditions whose
presence at some time-point is sufficient to change
the state of L at the next time-point
e.g., attack(X,Y) causes war(X,Y)

6. Crowdsourcing
“A strategy that combines the effort of the public to
solve a problem or produce a resource” (Wang et al.,
2013).
Games With A Purpose (GWAPs)
Genre of crowdsourcing
ESP Game, Verbocity, Common Consensus

7. “Knowledge Coder” Game
Output-agreement games template
Players required to agree on the same output they
produce
Game story
Planet Earth is captured by alien forces capable
of intercepting human communications in natural
language.
Join the resistance forces and encode human
knowledge in a form that is not readable by aliens.

8. Scoring and incentives
Players are rewarded with points:
For each successful mission attempt
When other players contribute and verify the
former players’ mission results and vice versa
Awards issued after a certain score

9. Considerations
Assumptions for human participants
Knowledgeable
Honest
Willing to participate
Game comprises multiple steps
Lower probability of user error
Easier control of the outcomes of each step
Facilitate the integration with knowledge
understanding systems

10. Considerations
Cheating: non-standard methods for creating an
advantage beyond normal gameplay
Anti-cheating mechanisms
Player anonymity
Internet address recording/filtering
Time-bounded missions

11. Acquiring Knowledge
Acquire broad knowledge
Use in Multiple Stories
Methodology comprises 6 steps casted as game
missions:
Mission 1 - Sentence processing
Mission 2 - Verb and noun identification
Mission 3 - Predicate construction
Mission 4 - Rule construction
Mission 5 - Rule generalization
Mission 6 - Rule evaluation

12. 1. Sentence processing
Sentence: A cat chased the mice.
After processing:
{cat,chase,mouse}

13. 2.Verb & noun identification
Selected phrase: {cat,chase,mouse}
After separation:
{cat,mouse} are nouns
{chase} is a verb

14. 3.Predicate construction
Formal expression: chase(cat,mouse)
Selected words: {cat,mouse} nouns,{chase} verb
action

15. 4.Rule construction
Rule 1: chase(cat,mouse) causes fear(mouse,cat)
Rule 2: chase(cat,mouse) implies can(cat,run)
Formal expression: chase(cat,mouse)

16. 5. Rule generalization
Rule 2: cat(X) and chase(X,Y) implies can(X,run)
Rule 1: chase(X,Y) implies can(X,run)
Rule: chase(cat,mouse) implies can(cat,run)

17. 6. Rule evaluation
Applicability
Sentence: A policeman was chasing a burglar.
Validity
Rule: chase(X,Y) implies can(X,run)

18. Rule Applicability
The conditions in the body of the rule are met in the
context of the selected sentence.
Sentence: A policeman was chasing a burglar.
Rule: chase(X,Y) implies can(X,run)
Body
X Y

19. Rule Validity
Decide whether the head of the rule follows from the
sentence.
Rule: chase(X,Y) implies can(X,run)
Head
Sentence: A policeman was chasing a burglar.
X

20. 5 participants
Men and women
18+
>High School education
2 Stories (Aesop's fables)
The Oxen and the
Butchers
The Doe and the Lion
Empirical Setting
Access to a test game deployment for 1 week
Training on how to play

21. Empirical Results
Number of generated rules: 93
Number of causality rules: 15
Number of implication rules: 78
Examples of generated rules
horn(X) and assemble(X) and carry(purpose) and
sharpen(X) and assemble(certain,X,carry(purpose))
implies have(ox,horns)
beast(X) and throw(Y,mouth,X) implies kill(X,Y)

22. Empirical Results
Examples of generated rules
beast(X) and man(Y) and doe(Z) and
exclaime(Z) and escape(Z,Y) and throw(Z,X)
implies kill(X,Z)
Example of a “good” rule
beast(X) and throw(Y,mouth,X) implies
kill(X,Y)
Typos are common
in GWAPs

23. Player Feedback
Missions 1 and 2
Easy to play
Informative instructions
Missions 3 and 4
Required time before players understand fully what
they were expected to do
Interesting
Mission 5
Not very challenging
Mission 6
Easy to play

24. Player Feedback
Interesting game story
Would advertise the game to their friends
Proposed tablet and mobile version of the game
Requested integration with social media

25. Conclusions & Future Work
Encouraging results in terms of the feasibility of our
methodology
Conduct further experiments with more stories and
players
Acquisition of not highly applicable rules
Need for stronger incentives to simplify the rules
Integrate “Knowledge Coder” with a reasoning engine
Framework based on psychologically-validated models
of narrative comprehension (Diakidoy et al., 2014)
Compare crowdsourcing methods used in “Knowledge
Coder” game with automated knowledge acquisition
methods

26. More information….
• Christos T. Rodosthenous
• Email: christos.rodosthenous@ouc.ac.cy
• WWW: http://cognition.ouc.ac.cy
Game is accessible online at:
http://cognition.ouc.ac.cy/narrative