Slides presented by Sebastian Riedel of UCL Machine Reading at an Iris. AI's meetup.
Slides cover the evolution of approaches to machine reading and describe a way to address the problem of working with very limited data.
4. Overview
What is Machine Reading?
How is it done?
In the past: knowledge and rules
Today: a lot of data and generic Deep Learning
But data is scarce
How do we get knowledge back in?
4
6. Machine Reading for Science …
6
“Which proteins interact with wild-type TES2?”
NF-kB is a protein
complex that controls
the transcription of
DNA. ... It plays a key
role in regulating the
immune response to
infection. ... Incorrect
regulation of NF-kB
has been linked to
many diseases and
disorders.
read
7. Machine Reading for Science …
7
“Which proteins interact with wild-type TES2?”
Reading Machine
“TRADD, …”
NF-kB is a protein
complex that controls
the transcription of
DNA. ... It plays a key
role in regulating the
immune response to
infection. ... Incorrect
regulation of NF-kB
has been linked to
many diseases and
disorders.
read
8. For Journalists …
8
“Who is in the upper echelon of the Iranian
Military?”
Reading Machine
“Ataollah Salehi, … “
TEHRAN — Iran’s
supreme leader, in an
unexpected move,
replaced the general
in charge of the
Iranian armed forces
on Tuesday with the
general’s deputy, a
member of the Islamic
Revolutionary Guards
Corps…
read
9. For Everyone
9
“What pension scheme should I use?”
Reading Machine
“A Self-Invested Personal Pension (SIPP)”
It makes sense to put
some money away for
when you’re older and
that’s what pension
schemes help you do.
You save a little of
your income regularly
during your working
life so you can have
an income in later life,
when you want to
work less or retire.
read
11. The Reading Machine User
11
NF-kB is a protein
complex that controls
the transcription of
DNA. ... It plays a key
role in regulating the
immune response to
infection. ... Incorrect
regulation of NF-kB
has been linked to
many diseases and
disorders.
Reading Machine
“Which proteins interact with wild-type TES2?”
“TRADD, …”
12. And the Developer
12
“Which proteins interact with wild-type TES2?”
“TRADD, …”
Reading Machine
“I want to build
the best
reading machine!”
14. And the Developer
14
Reading Machine
1960
1990
“I have a lot of knowledge about language!”
If Protein A is the syntactic
subject of an “activates” verb
and Protein B is the object,
and …
then
A interacts with B
…
“I want to build
the best
reading machine!”
15. And the Developer
Reading Machine
1990
“I have a lot of knowledge about language!”
2014
15
“I want to build the best reading machine!”
“I have some knowledge about language”
“I also have some data though!”
“I want to build
the best
reading machine!”
distance < 10
“activates” in between
16. And the Developer
Reading Machine
2014
16
“I have some knowledge about language”
“I also have some data though!”
“I have no knowledge about language*”
“But I have a high capacity learner…”
*“rather: I don’t want to bother”
“I want to build
the best
reading machine!”
Encoder
Decoder
“and lots of data!”
Generic Recurrent Neural Network
18. Reading Machine
2014
18
“I have some knowledge about language”
“I also have some data though!”
“I have no knowledge about language*”
“But I have a high capacity learner…”
“I want to build
the best
reading machine!”
Encoder
Decoder
“and lots of data!”
2015
19. Reading Machine
19
“I have no knowledge about language*”
“But I have a high capacity learner…”
Encoder
Decoder
“and lots of data!”
2015
“I want to build
the best
reading machine!”
"Cody ran to Melanie ' s farm. The
distance is 18 yards from Cody ' s farm to
Melanie ' s farm. It took Cody 2 hours to
get there. How fast did Cody go?” 18/2
~ 200 Training instances
Math Word Problems
20. Math Word Problems
20
“I have very little data!”
A has X items, (gives away | gets) Y
items. How many does (he|she) have
now? X (+|-) Y
“But I know something!”
"Cody ran to Melanie ' s farm. The
distance is 18 yards from Cody ' s farm to
Melanie ' s farm. It took Cody 2 hours to
get there. How fast did Cody go?” 18/2
Reading Machine
Encoder
Decoder
21. Learning to Generate Training Data
21
Reading
Machine
generates data for
to do well on
A has X items, (gives away | gets) Y
items. How many does (he|she) have
now? X (+|-) Y
Data Generation Model
“But I know something!”
"Cody ran to Melanie ' s farm. The
distance is 18 yards from Cody ' s farm to
Melanie ' s farm. It took Cody 2 hours to
get there. How fast did Cody go?” 18/2
“I have very little data!”
defines
Bouchard & Stenetorp, EMNLP 2016
22. How to Generate What?
22
A has X items, (gives away | gets) Y
items. How many does (he|she) have
now? X (+|-) Y
67
69.75
72.5
75.25
78
Prev. Work Data Knowledge K+D
“I have very little data!”
"Cody ran to Melanie ' s farm. The
distance is 18 yards from Cody ' s farm to
Melanie ' s farm. It took Cody 2 hours to
get there. How fast did Cody go?” 18/2
“But I know something!”
23. Information Extraction
23
Sebastian is a Reader at UCL
Sebastian lives in London
Who does Sebastian work for?
UCL
“Readers are University employees”
“But I know something!”
“I have very little data!”
Reading Machine
24. Regularisation
24
“Readers are University employees”
“But I know something!”
Knowledge Loss
optimise on
optimise on
convert
Sebastian is a Reader at UCL
Sebastian lives in London
Who does Sebastian work for?
UCL
“I have very little data!”
Reading Machine
Rocktäschel, Demeester, Singh, NAACL 2015, EMNLP 2016
25. Regularisation
25
“Readers are University employees”
“But I know something!”
0
15
30
45
60
Data Knowledge K+D Ours
Sebastian is a Reader at UCL
Sebastian lives in London
Who does Sebastian work for?
UCL
“I have very little data!”
26. Learning to Program
26
Sort animal, pug, dog, mammal
animal, mammal, dog, pub
The program is recursive, and uses
an (unknown) comparison function
“But I know something!”
“I have very little data!”
Machine
27. Learning to Program
27
Sort animal, pug, dog, mammal
animal, mammal, dog, pub
def sort(input):
???
compare(???, ???)
???
sort(???(input))
“But I know something!”
“I have very little data!”
Machine
28. Knowledge Compilation
28
Sort animal, pug, dog, mammal
animal, mammal, dog, pub
def sort(input):
???
compare(???, ???)
???
sort(???(input))
“But I know something!”
Machine
compile to
optimise on
“I have very little data!”
Bosnjak, Rocktäschel, 2016, arxiv
29. Knowledge Compilation
29
Sort animal, pug, dog, mammal
animal, mammal, dog, pub
“And I have very little data!”
Machine
optimise on
A Neural Program Trace:
30. Summary
Machine Reading today:
high capacity deep learner & a lot of data
But Machine Reading reality: Not much data
How to inject knowledge into deep learning?
By (generating) data
By regularising model
By compiling knowledge into model structure
30
