TensorFlow 深度學習快速上手班--自然語言處理應用
•
17 likes•2,487 views
http://www.hcinnovation.tw/tc/index.php/k2-blog/item/66-tensorflow 課程講義4
Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Viewers also liked
Viewers also liked (20)
Similar to TensorFlow 深度學習快速上手班--自然語言處理應用
Similar to TensorFlow 深度學習快速上手班--自然語言處理應用 (20)
More from Mark Chang
More from Mark Chang (13)
Recently uploaded
Recently uploaded (20)
TensorFlow 深度學習快速上手班--自然語言處理應用
- 2. • ⾃自然語⾔言處理簡介 • Word2vec神經網路 • 語意運算實作
- 3. ⾃自然語⾔言處理簡介
- 4. ⾃自然語⾔言處理 • ⾃自然語⾔言處理是⼈人⼯工智慧和語⾔言學領域的分⽀支 – 探討如何處理及運⽤用⾃自然語⾔言 • ⾃自然語⾔言理解系統 – 把⾃自然語⾔言轉化為電腦易於處理的形式。 • ⾃自然語⾔言⽣生成系統 – 把電腦程式數據轉化為⾃自然語⾔言。 • https://zh.wikipedia.org/wiki/%E8%87%AA %E7%84%B6%E8%AF%AD %E8%A8%80%E5%A4%84%E7%90%86
- 10. Word2vec神經網路
- 11. ⽂文字的語意 • 某個字的語意，可從它的上下⽂文得知 dog 和 cat 語意相近. The dog run. A cat run. A dog sleep. The cat sleep. A dog bark. The cat meows.
- 12. 語意向量 The dog run. A cat run. A dog sleep. The cat sleep. A dog bark. The cat meows. the a run sleep bark meow dog 1 2 2 2 1 0 cat 2 1 2 2 0 1
- 13. 語意向量 dog (1, 2,..., xn) cat (2, 1,..., xn) Car (0, 0,..., xn)
- 14. 語意向量相似度 • A 和 B 的Cosine Similarity 為： A · B |A||B| dog (a1,a2,...,an) cat (b1,b2,...,bn) dog 和 cat 的cosine similarity為： a1b1 + a2b2 + ... + anbn p a2 1 + a2 2 + ... + a2 n p b2 1 + b2 2 + ... + b2 n
- 15. 語意向量加減運算 Woman + King - Man = Queen Woman Queen Man King King - Man King - Man
- 16. 語意向量維度太⼤大 (x1=the, x2 =a,..., xn) dog 語意向量的維度等於總字彙量 x 1 x 2 x 3 x 4 xn...
- 18. One-Hot Encoding dog cat run fly １ ０ ０ ０ ０ １ ０ ０ ０ ０ １ ０ ０ ０ ０ １
- 19. Initialize Weights dog cat run fly dog cat run fly W = 2 6 6 4 w11 w12 w13 w21 w22 w23 w31 w32 w33 w31 w32 w43 3 7 7 5V = 2 6 6 4 v11 v12 v13 v21 v22 v23 v31 v32 v33 v31 v32 v43 3 7 7 5
- 21. Compressed Vectors dog cat run fly v11 v12 v13 v21 v22 v23 w31 w32 w33 w41 w42 w43 dog cat run fly dog cat run fly
- 22. Context Word dog １ ０ ０ ０ v11 v12 v13 v11 v12 v13 run ０ ０ １ ０ w31 w32 w33 dog cat run fly dog cat run fly 1 1 + e V1W3 ⇡ 1 V1 · W3 = v11w31 + v12w32 + v13w33
- 23. Context Word cat １ ０ ０ ０ v11 v12 v13 v21 v22 v23 run ０ ０ １ ０ w31 w32 w33 dog cat run fly V2 · W3 = v21w31 + v22w32 + v23w33 dog cat run fly 1 1 + e V2W3 ⇡ 1
- 24. Non-context Word dog １ ０ ０ ０ v11 v12 v13 v11 v12 v13 fly ０ ０ １ ０ w41 w42 w43 V1 · W4 = v11w41 + v12w42 + v13w43 1 1 + e V1W4 ⇡ 0 dog cat run fly dog cat run fly
- 25. Non-context Word cat １ ０ ０ ０ v11 v12 v13 v21 v22 v23 w41 w42 w43 ０ ０ １ ０ V2 · W4 = v21w41 + v22w42 + v23w43 dog cat run fly dog cat run fly fly 1 1 + e V2W4 ⇡ 0
- 26. Result dog cat run fly dog cat run fly v11 v12 v13 v21 v22 v23 w31 w32 w33 w41 w42 w43 dog cat run fly
- 27. 語意運算實作
- 28. 語意運算實作 https://github.com/ckmarkoh/ntc_deeplearning_tensorflow/ blob/master/sec4/semantics.ipynb
- 29. 訓練資料 anarchism originated as a term of abuse first used against early working class radicals including the diggers of the english revolution and the sans culottes of the french revolution whilst the term is still used in a pejorative way to describe any act that used violent means to destroy the organization of society it has also been taken up as a positive label by self defined anarchists the word anarchism is derived from the greek without archons ruler chief king anarchism as a political philosophy is the belief that rulers are unnecessary and should be abolished although there are differing interpretations of what this means anarchism also refers to related social movements that advocate the elimination of authoritarian institutions particularly the state the word anarchy as most anarchists use it does not imply chaos nihilism or anomie but rather a harmonious anti authoritarian society in place of what
- 30. 前處理 anarchism originated as a term of abuse first used against early working class radicals including the diggers of the english revolution and the sans culottes of the french revolution whilst the term is still used in a pejorative way to describe any act that used violent means to destroy the organization of society it has also been taken up …. [‘anarchism’, ‘originated’, ‘as’, ‘a’, ‘term’, ‘of’, ‘abuse’, ‘first’, ‘used’, ‘against’, ‘early’, ‘working’, ‘class’, ‘radicals’, ‘including’, ‘the’, ‘diggers’, ‘of’, ‘the’, ‘english’, ‘revolution’, ‘and’, ‘the’, ‘sans’, ‘culottes’, ‘of’, ‘the’, ‘french’, ‘revolution’, ‘whilst’, ‘the’, ‘term’, ‘is’, ‘still’, ‘used’, ‘in’, ‘a’, ‘pejorative’, ‘way’, ‘to’, ‘describe’, ‘any’, ‘act’, ‘that’, ‘used’, ‘violent’, ‘means’, ‘to’, ‘destroy’, ‘the’... ]
- 31. 前處理 ‘the’, ‘english’, ‘revolution’, ‘and’, ‘the’, ‘sans’, UNK, 'of', 'the', 'french', 'revolution’… 1, 103, 855, 3, 1, 15068, 0, 2, 1, 151, 855, … ‘the’, ‘english’, ‘revolution’, ‘and’, ‘the’, ‘sans’, ‘culottes’, 'of', 'the', 'french', 'revolution’… ‘the’, ‘english’, ‘revolution’, ‘and’, ‘the’, ‘sans’, ‘culottes’, 'of', 'the', 'french', 'revolution’… 字典外的字， 用UNK代替。 將字轉換成字 典內的代碼。 根據詞頻， 轉換成字典 {“UNK”: 0, “the”: 1, “of”: 2, “and”: 3, “one”: 4, “in”: 5, “a”: 6, “to”: 7, “zero”: 8, “nine”: 9, .... } # 字典大小 vocabulary_size = 50000
- 32. 前處理 5239, 3084, 12, 6, 195, 2, 3137, 46, 59, 156, 128, 742, 477, 10572, 134, 1, 27549, 2, 1, 103, 855, 3, 1, 15068, 0, 2, 1, 151, 855, … input output 3084 5239 3084 12 12 3084 12 6 6 12 6 195 195 6 195 2 3084 5239 word2vec
- 33. 前處理 5239, 3084, 12, 6, 195, 2, 3137, 46, 59, 156, 128, 742, 477, 10572, 134, 1, 27549, 2, 1, 103, 855, 3, 1, 15068, 0, 2, 1, 151, 855, … generate_batch(batch_size=8, num_skips=2, skip_window=1) batch size input 3084 3084 12 12 6 6 195 195 output 5239 12 3084 6 12 195 6 2 num_skips batch_size skip_window=1
- 34. Computational Graph train_inputs = tf.placeholder(tf.int32, shape=[batch_size]) train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1]) with tf.device('/cpu:0'): embeddings = tf.Variable( tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0)) embed = tf.nn.embedding_lookup(embeddings, train_inputs) nce_weights = tf.Variable( tf.truncated_normal([vocabulary_size, embedding_size], stddev=1.0 / math.sqrt(embedding_size))) nce_biases = tf.Variable(tf.zeros([vocabulary_size])) loss = tf.reduce_mean( tf.nn.nce_loss(nce_weights, nce_biases, embed, train_labels, num_sampled, vocabulary_size)) optimizer = tf.train.GradientDescentOptimizer(1.0).minimize(loss)
- 35. Device with tf.device('/cpu:0’) 在CPU上執行以下定義的Computational Graph 由於Tensorflow未支援 embedding_lookup 在GPU上 執行，故需令它在CPU上執行。
- 36. Inputs Outputs word2vec train_inputs = tf.placeholder(tf.int32, shape=[batch_size]) train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1]) train_inputs 3084 3084 12 12 6 6 195 195 train_labels 5239 12 3084 6 12 195 6 2
- 37. Embedding Lookup embeddings = tf.Variable(tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0)) embed = tf.nn.embedding_lookup(embeddings, train_inputs) train_inputs 2 embeddings embedding_lookup
- 38. NCE Weights • NCE: Noise Contrastive Estimation nce_weights = tf.Variable( tf.truncated_normal([vocabulary_size, embedding_size], stddev=1.0 / math.sqrt(embedding_size) )) nce_biases = tf.Variable( tf.zeros([vocabulary_size]) ) nce_weights nce_biases
- 39. NCE Loss loss = tf.reduce_mean( tf.nn.nce_loss(nce_weights, nce_biases, embed, train_labels, num_sampled, vocabulary_size)) １ ０ ０ ０ v11 v12 v13 v21 v22 v23 ０ ０ １ ０ w31 w32 w33 1 1 + e V2W3 ⇡ 1 １ ０ ０ ０ v11 v12 v13 v21 v22 v23 w41 w42 w43 ０ ０ １ ０ 1 1 + e V2W4 ⇡ 0 Positive Negative cost = log( 1 1 + e vT I wpos ) + X neg log(1 1 1 + e vT I wneg )
- 40. Train feed_dict = {train_inputs: batch_inputs, train_labels: batch_labels} _, loss_val = session.run([optimizer, loss], feed_dict=feed_dict) loss_val batch_inputs 3084 3084 12 12 6 6 195 195 batch_labels 5239 12 3084 6 12 195 6 2
- 41. Result final_embeddings array([[-0.02782757, -0.16879494, -0.06111901, ..., -0.25700757, -0.07137159, 0.0191142 ], [-0.00155336, -0.00928817, -0.0535327 , ..., -0.23261793, -0.13980433, 0.18055709], [ 0.02576068, -0.06805354, -0.03688766, ..., -0.15378961, 0.00459271, 0.0717089 ], ..., [ 0.01061165, -0.09820389, -0.09913248, ..., 0.00818674, -0.12992384, 0.05826835], [ 0.0849214 , -0.14137401, 0.09674817, ..., 0.04111136, -0.05420518, -0.01920278], [ 0.08318492, -0.08202577, 0.11284919, ..., 0.03887166, 0.01556483, 0.12496017]], dtype=float32)
- 42. Visualization
- 43. Most Similar Words def get_most_similar(word, top=10): wid = dictionary.get(word,-1) result = np.dot(final_embeddings[wid:wid+1,:],final_embeddings.T) result = result [0].argsort().tolist() result.reverse() for idx in result [:10]: print(reverse_dictionary[idx]) get_most_similar(one) one six two four seven three ...
- 44. 講師資訊 • Email: ckmarkoh at gmail dot com • Blog: http://cpmarkchang.logdown.com • Github: https://github.com/ckmarkoh Mark Chang • Facebook: https://www.facebook.com/ckmarkoh.chang • Slideshare: http://www.slideshare.net/ckmarkohchang • Linkedin: https://www.linkedin.com/pub/mark-chang/85/25b/847 44