3. “Unfortunately, this turns out to be very difficult.
Very very difficult. And given the tremendous
promise, there are lots of people working on”
--- Chris Olah
4. "Validating correctness is a difficult enterprise
because the system is inherently stochastic and
only intended to behave in a certain way in
expectation — potentially after hours of
computation.”
--- TensorFlow White Paper
5. Agenda
o Watson API
n Watson vs. Watson
n Natural Language Classifier
n Dialog
n Cortana, Alexa との比較
n Retrieve and Rank
o Deep Learningでの自然言語へのアプローチ
n Deep Learning, NLP, and Representations
6. Agenda
o Google TensolFlowの登場
n Google TensolFlowとは何か?
n TensorFlowのプログラミング・モデルと
基本的なコンセプト
n TensorFlowプログラム サンプル1
線形回帰
n TensorFlowプログラム サンプル 2
手書き文字の認識
o TensolFlow White Paper (翻訳)
23. Natural Language Classifier
Demo
o Is it hot outside?
o Natural Language Classifier is 98% confident
that the question submitted is talking about
'temperature’.
Classification: temperature
Confidence: 98%
http://goo.gl/hGByud
24. Natural Language Classifier
Demo
o What is the expected high for today?
o Natural Language Classifier is 98% confident
that the question submitted is talking about
'temperature’.
Classification: temperature
Confidence: 98%
http://goo.gl/hGByud
25. Natural Language Classifier
Demo
o Will it be foggy tomorrow morning?
o Natural Language Classifier is 68% confident
that the question submitted is talking about
‘conditions’.
Classification: conditions
Confidence: 68%
http://goo.gl/hGByud
26. Natural Language Classifier
Demo
o Will it be foggy tomorrow morning?
o Natural Language Classifier is 68% confident
that the question submitted is talking about
‘conditions’.
Classification: conditions
Confidence: 68%
http://goo.gl/hGByud
27. Natural Language Classifier
Demo
o Will it be foggy tomorrow morning?
o Natural Language Classifier is 68% confident
that the question submitted is talking about
‘conditions’.
Classification: conditions
Confidence: 68%
http://goo.gl/hGByud
28. Natural Language Classifier
Demo
o Should I prepare for sleet?
o Natural Language Classifier is 91% confident
that the question submitted is talking about
‘conditions’.
Classification: conditions
Confidence: 91%
http://goo.gl/hGByud
29. Natural Language Classifier
Demo
o Will there be a storm today?
o Natural Language Classifier is 98% confident
that the question submitted is talking about
‘conditions’.
Classification: conditions
Confidence: 98%
http://goo.gl/hGByud
55. Grammerタグ内の Wild Card 文字
Asterisk (*)
o “What is * dialog” は、次のものにマッチする
o "What is a dialog"
o "What is this thing called a dialog"
o “What is the meaning of a dialog”
o ...
56. Grammerタグ内の Wild Card 文字
Dollar sign ($)
o “$ is a dialog” は、次のものにマッチする。
o "So, can you tell me what is a dialog, please"
o "I've heard about virtual agents, but is a dialog
one of those"
o "Tell me, is a dialog something I can create"
o "Is a dialog something I can use for my
business”
o ...
57. Grammerタグ内の Wild Card 文字
Percent sign (%)
o “% is a dialog”は、次のものにマッチする。
o What is a dialog
o Can you tell me about a dialog
o What is it
o is
o a dialog
o ...
58. Grammerタグ内の Wild Card 文字
Hash sign (#)
o “# is dialog”は、次のものにマッチする。
o What is a Dialog
o Can you tell me what Dialog is
o Is this thing I'm looking at a Dialog
o What is Dialog
62. MS Cortana
Voice Commandの定義サンプル
<VoiceCommands
xmlns="http://schemas.microsoft.com/voicecommands/1.2">
<CommandSet xml:lang="en-us”
Name="AdventureWorksCommandSet_en-us">
<CommandPrefix> Adventure Works, </CommandPrefix>
<Example> Show trip to London </Example>
<Command Name="showTripToDestination">
<Example> Show trip to London </Example>
<ListenFor RequireAppName="BeforeOrAfterPhrase">
show [my] trip to {destination}
</ListenFor>
<ListenFor RequireAppName="ExplicitlySpecified">
show [my] {builtin:AppName} trip to {destination}
</ListenFor>
<Feedback> Showing trip to {destination} </Feedback>
<Navigate Target="foo"/>
</Command>
64. helloworld.speechAsset.
SampleUtterances.txt
HelloWorldIntent say hello
HelloWorldIntent say hello world
HelloWorldIntent hello
HelloWorldIntent say hi
HelloWorldIntent say hi world
HelloWorldIntent hi
HelloWorldIntent how are you
HelpIntent help
HelpIntent help me
HelpIntent what can I ask you
HelpIntent get help
HelpIntent to help
HelpIntent to help me
intentName 発話
cortanaのListenFor
と同じ働きをする。それぞ
れの発話に、intent
Nameが付けられている。
“say hello”と“how are
you”は、同じintent
Nameが割り当てられて
いる。
87. “An example of what recurrent neural nets
can now do” WikiPedia で学習したもの
o The meaning of life is the tradition of the
ancient human reproduction: it is less
favorable to the good boy for when to remove
her bigger. In the show’s agreement
unanimously resurfaced. The wild pasteured
with consistent street forests were
incorporated by the 15th century BE. In 1996
the primary rapford undergoes an effort that
the reserve conditioning, written into Jewish
cities, sleepers to incorporate the .St Eurasia
that activates the population.
http://goo.gl/vHRHSn
88. “An example of what recurrent neural nets
can now do” New York Timesで学習
o while he was giving attention to the second
advantage of school building a 2-for-2 stool
killed by the Cultures saddled with a halfsuit
defending the Bharatiya Fernall ’s office . Ms .
Claire Parters will also have a history temple
for him to raise jobs until naked Prodiena to
paint baseball partners , provided people to
ride both of Manhattan in 1978 , but what was
largely directed to China in 1946 , focusing on
the trademark period is the sailboat yesterday
and comments on whom they obtain overheard
within the 120th anniversary , where ......
http://goo.gl/vHRHSn
90. Word2Vecの登場
o 2013年に、GoogleのTomas Mikolovらは、語が埋め
込まれたベクター空間が、言語学的に(文法的にも、意味
論的にも)面白い性質を持っていることを発見する。
o “We find that these representations are
surprisingly good at capturing syntactic and
semantic regularities in language, and that
each relationship is characterized by a relation-
specific vector offset.” (Linguistic Regularities
in Continuous Space WordGoog
Representations
http://research.microsoft.com/pubs/189726/rvecs.pdf )
o Jeff Deanも、この論文に注目して、共同の作業が始まる。
115. 文 “This film does’nt care about cleverness wit or
any other kind of intelligenct humor” の解析
Socher, et al. http://goo.gl/bPAQ68
文法構造をニューラル・
ネットで表現する!
123. TensorFlowのコードとグラフのサンプル
import tensorflow as tf
b = tf.Variable(tf.zeros([100]))
W = tf.Variable(tf.random_uniform([784,100],-1,1))
x = tf.placeholder(name=“x”)
relu = tf.nn.relu(tf.matmul(W, x) + b)
C = [...]
s = tf.Session()
for step in xrange(0, 10):
input = ...construct 100-D input array ...
result = s.run(C, feed_dict={x: input})
print step, result
134. ....
W = tf.Variable(tf.random_uniform([1], -1.0, 1.0))
b = tf.Variable(tf.zeros([1]))
y = W * x_data + b
# Minimize the mean squared errors.
loss = tf.reduce_mean(tf.square(y - y_data))
optimizer = tf.train.GradientDescentOptimizer(0.5)
train = optimizer.minimize(loss)
# Before starting, initialize the variables. We will 'run' this first.
init = tf.initialize_all_variables()
# Launch the graph.
sess = tf.Session()
sess.run(init)
# Fit the line.
for step in xrange(201):
sess.run(train)
if step % 20 == 0:
print(step, sess.run(W), sess.run(b))
135. ....
W = tf.Variable(tf.random_uniform([1], -1.0, 1.0))
b = tf.Variable(tf.zeros([1]))
y = W * x_data + b
# Minimize the mean squared errors.
loss = tf.reduce_mean(tf.square(y - y_data))
optimizer = tf.train.GradientDescentOptimizer(0.5)
train = optimizer.minimize(loss)
# Before starting, initialize the variables. We will 'run' this first.
init = tf.initialize_all_variables()
# Launch the graph.
sess = tf.Session()
sess.run(init)
# Fit the line.
for step in xrange(201):
sess.run(train)
if step % 20 == 0:
print(step, sess.run(W), sess.run(b))
グラフの定義
アルゴリズム
グラフの起動
グラフで訓練
繰り返し
158. If α is too small, gradient
descent can be slow.
If α is too large, gradient
descent can overshoot the
minimum. It may fail to
converge, or even diverge.
アルファが、あまりに小さいと、
Gradient descentは、ゆっくり
にしか進まない。
アルファが、あまりに大きいと、
Gradient descentは、最小値
を通り越してしまう。場合によっ
ては、振動したり、収束しない場
合もある。
このアルファを学習率
(Learning Rate)という。
163. ....
W = tf.Variable(tf.random_uniform([1], -1.0, 1.0))
b = tf.Variable(tf.zeros([1]))
y = W * x_data + b
# Minimize the mean squared errors.
loss = tf.reduce_mean(tf.square(y - y_data))
optimizer = tf.train.GradientDescentOptimizer(0.5)
train = optimizer.minimize(loss)
# Before starting, initialize the variables. We will 'run' this first.
init = tf.initialize_all_variables()
# Launch the graph.
sess = tf.Session()
sess.run(init)
# Fit the line.
for step in xrange(201):
sess.run(train)
if step % 20 == 0:
print(step, sess.run(W), sess.run(b))
165. ....
# Create the model
x = tf.placeholder("float", [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
# Define loss and optimizer
y_ = tf.placeholder("float", [None, 10])
cross_entropy = -tf.reduce_sum(y_ * tf.log(y))
train_step = tf.train.GradientDescentOptimizer(0.01)¥
.minimize(cross_entropy)
# Train
tf.initialize_all_variables().run()
for i in range(1000):
batch_xs, batch_ys = mnist.train.next_batch(100)
train_step.run({x: batch_xs, y_: batch_ys})
ソースの全体は、こちらにある。
https://goo.gl/MwscZO
166. ....
# Create the model
x = tf.placeholder("float", [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
# Define loss and optimizer
y_ = tf.placeholder("float", [None, 10])
cross_entropy = -tf.reduce_sum(y_ * tf.log(y))
train_step = tf.train.GradientDescentOptimizer(0.01)¥
.minimize(cross_entropy)
# Train
tf.initialize_all_variables().run()
for i in range(1000):
batch_xs, batch_ys = mnist.train.next_batch(100)
train_step.run({x: batch_xs, y_: batch_ys})
グラフの定義
アルゴリズム
グラフで訓練
繰り返し
ソースの全体は、こちらにある。
https://goo.gl/MwscZO
167. 線形回帰とクラス分類との比較
変数の定義部分
W = tf.Variable(tf.random_uniform([1], -1.0, 1.0))
b = tf.Variable(tf.zeros([1]))
y = W * x_data + b
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
実数が行列に変わっているだけで、ほとんど同じものだ。
mutmul(x, W)は、行列同士の掛け算である。
クラス分類の方に、softmaxがつくのは当然である。
線形回帰
クラス分類
190. TensorFlowのコードとグラフのサンプル
import tensorflow as tf
b = tf.Variable(tf.zeros([100]))
W = tf.Variable(tf.random_uniform([784,100],-1,1))
x = tf.placeholder(name=“x”)
relu = tf.nn.relu(tf.matmul(W, x) + b)
C = [...]
s = tf.Session()
for step in xrange(0, 10):
input = ...construct 100-D input array ...
result = s.run(C, feed_dict={x: input})
print step, result