Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Similar to BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (20)
More from gohyunwoong
Recently uploaded
Recently uploaded (20)
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- 1. Transformer & BERT Review by Hyunwoong (github.com/gusdnd852) 1
- 2. Presentation Outline 1. The Transformer Network (Backbone of BERT) 1.1 Attention Mechanism (Seq2Seq with Attention) 1.2 Transformer Architecture (Transformer) 2. Background : emergence of BERT (Introduction & Related Work) 2.1 Word Embedding (Word Representation in ML) 2.2 Embedding from Language Model (ELMo) 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 3. BERT & Experiment (Proposed Method & Experimental Result) 2
- 3. 1.1 Attention Mechanism (Seq2Seq with Attention) 3
- 4. 4 1.1 Attention Mechanism (Seq2Seq with Attention) There are plain Seq2Seq model Reference : Raimi Karim
- 5. 5 1.1 Attention Mechanism (Seq2Seq with Attention) If sequence is long, it shows very bad performance But, Why? Reference : Raimi Karim
- 6. 6 1.1 Attention Mechanism (Seq2Seq with Attention) Context vector size is fixed, we can not put every information Reference : Raimi Karim
- 7. 7 1.1 Attention Mechanism (Seq2Seq with Attention) Network confused. “so many information in small vector. which part is important?” Reference : Raimi Karim
- 8. 8 1.1 Attention Mechanism (Seq2Seq with Attention) So, We will use output of each time step Reference : Raimi Karim
- 9. 9 1.1 Attention Mechanism (Seq2Seq with Attention) Firstly, we can get the encoder hidden state Reference : Raimi Karim
- 10. 10 1.1 Attention Mechanism (Seq2Seq with Attention) And, compute dot product of encoder hidden state and output of decoder(firstly, BOS’s output) to get relationship of words Reference : Raimi Karim
- 11. 11 1.1 Attention Mechanism (Seq2Seq with Attention) We call this framework as Query, Key, Value Reference : Raimi Karim
- 12. 12 1.1 Attention Mechanism (Seq2Seq with Attention) Query Key Key Key Key Reference : Raimi Karim
- 13. 13 1.1 Attention Mechanism (Seq2Seq with Attention) Query Key Key Key Key And compute softmax score to adjust dot product to [0,1] Reference : Raimi Karim
- 14. 14 1.1 Attention Mechanism (Seq2Seq with Attention) Multiply score with keys (hidden state of encoders) Query Key Key Key Key Value Value Value Value Reference : Raimi Karim
- 15. 15 1.1 Attention Mechanism (Seq2Seq with Attention) Add Every Value. Now we call this Align (Attention) Query Key Key Key Key Value Value Value Value Align (attention) Reference : Raimi Karim
- 16. 16 1.1 Attention Mechanism (Seq2Seq with Attention) Align (Attention) is used as input Reference : Raimi Karim
- 17. 17 1.1 Attention Mechanism (Seq2Seq with Attention) Align (Attention) is used as input NMT by jointly learning and align (2014) Reference : Raimi Karim
- 18. 18 1.1 Attention Mechanism (Seq2Seq with Attention) Effective Approaches to Attention-based NMT (2015) Reference : Raimi Karim Align (Attention) is used as input
- 20. 20 1.2 Transformer Architecture 1. Parallelism (Recurrent to Pos Encoding) 2. Consider self attention (not only enc-dec attention)
- 21. 21 1.2 Transformer Architecture There are <s></s><PAD><UNK> tokens (max_len = 6) Reference : Kim Dong Hwa Input data format.
- 22. 22 1.2 Transformer Architecture Data size is like this (zero initialized) d_model = 4, max_len = 6 I : [0, 0, 0, 0] am : [0, 0, 0, 0] kim : [0, 0, 0, 0] <PAD> : [0, 0, 0, 0] <PAD> : [0, 0, 0, 0] <PAD> : [0, 0, 0, 0] Reference : Kim Dong Hwa Input data format. data = torch.zeros(batch_size, max_len, d_model) size = data.size() # [?, 6, 4] ? Is batchsize (number of sentence in here)
- 23. 23 1.2 Transformer Architecture nn.Embedding(vocab_size, d_model) work like this Reference : Kim Dong Hwa Input Embedding
- 24. 24 1.2 Transformer Architecture Pos Encoding – 1 (Weight Matrix) Reference : Kim Dong Hwa Positional Encoding
- 25. 25 1.2 Transformer Architecture Pos Encoding – 2 (Sinusoid Function) Reference : Kim Dong Hwa Positional Encoding
- 26. 26 1.2 Transformer Architecture Reference : Kim Dong Hwa Embedding Dropout
- 27. 27 1.2 Transformer Architecture Reference : Kim Dong Hwa Multi-Head Attention
- 28. 28 1.2 Transformer Architecture Reference : Kim Dong Hwa Multi-Head Attention
- 29. 29 1.2 Transformer Architecture Reference : Kim Dong Hwa Multi-Head Attention
- 30. 30 1.2 Transformer Architecture Reference : Kim Dong Hwa Multi-Head Attention
- 31. 31 1.2 Transformer Architecture Reference : Kim Dong Hwa Multi-Head Attention (Apply Mask)
- 32. 32 1.2 Transformer Architecture Reference : Kim Dong Hwa Multi-Head Attention (Apply Mask)
- 33. 33 1.2 Transformer Architecture Reference : Kim Dong Hwa Multi-Head Attention (Apply Mask)
- 34. 34 1.2 Transformer Architecture Reference : Kim Dong Hwa Multi-Head Attention
- 35. 35 1.2 Transformer Architecture Reference : Kim Dong Hwa Multi-Head Attention
- 36. 36 1.2 Transformer Architecture Reference : Kim Dong Hwa Add & Norm
- 37. 37 1.2 Transformer Architecture Reference : Kim Dong Hwa Add & Norm
- 38. 38 1.2 Transformer Architecture Reference : Kim Dong Hwa Add & Norm
- 39. 39 1.2 Transformer Architecture Reference : Kim Dong Hwa Positionwise Feed Foward
- 40. 40 1.2 Transformer Architecture Reference : Kim Dong Hwa Positionwise Feed Foward
- 41. 41 1.2 Transformer Architecture Reference : Kim Dong Hwa Add & Norm
- 42. 42 1.2 Transformer Architecture Reference : Kim Dong Hwa Decoder
- 43. 43 1.2 Transformer Architecture Reference : Kim Dong Hwa Masked Multi-Head Attention
- 44. 44 1.2 Transformer Architecture Reference : Kim Dong Hwa Masked Multi-Head Attention
- 45. 45 1.2 Transformer Architecture Reference : Kim Dong Hwa Masked Multi-Head Attention
- 46. 46 1.2 Transformer Architecture Reference : Kim Dong Hwa Masked Multi-Head Attention
- 47. 47 1.2 Transformer Architecture Reference : Kim Dong Hwa Masked Multi-Head Attention (Apply Mask)
- 48. 48 1.2 Transformer Architecture Reference : Kim Dong Hwa Masked Multi-Head Attention (Apply Mask)
- 49. 49 1.2 Transformer Architecture Reference : Kim Dong Hwa Masked Multi-Head Attention
- 50. 50 1.2 Transformer Architecture Reference : Kim Dong Hwa Add & Norm
- 51. 51 1.2 Transformer Architecture Reference : Kim Dong Hwa Encoder–Decoder Multi-Head Attention
- 52. 52 1.2 Transformer Architecture Reference : Kim Dong Hwa Encoder–Decoder Multi-Head Attention
- 53. 53 1.2 Transformer Architecture Reference : Kim Dong Hwa Encoder–Decoder Multi-Head Attention
- 54. 54 1.2 Transformer Architecture Reference : Kim Dong Hwa Encoder–Decoder Multi-Head Attention
- 55. 55 1.2 Transformer Architecture Reference : Kim Dong Hwa Encoder–Decoder Multi-Head Attention
- 56. 56 1.2 Transformer Architecture Reference : Kim Dong Hwa Add & Norm
- 57. 57 1.2 Transformer Architecture Reference : Kim Dong Hwa Positionwise Feed Foward
- 58. 58 1.2 Transformer Architecture Reference : Kim Dong Hwa Positionwise Feed Foward
- 59. 59 1.2 Transformer Architecture Reference : Kim Dong Hwa Add & Norm
- 60. 60 1.2 Transformer Architecture Reference : Kim Dong Hwa Linear & Softmax
- 61. 61 1.2 Transformer Architecture Reference : Kim Dong Hwa Summary
- 62. 2.1 Word Embedding (Word Representation in ML) 62
- 63. 2.1 Word Embedding (Word Representation in ML) 63 How to represent word in computer ?
- 64. 2.1 Word Embedding (Word Representation in ML) 64 … We Want Dense Representation !! Sparse Representation (One-Hot)
- 65. 2.1 Word Embedding (Word Representation in ML) 65 Dense Representation (Word2Vec) CBOWSkip Gram
- 66. 2.1 Word Embedding (Word Representation in ML) 66 Dense Representation (Word2Vec) CBOWSkip Gram
- 67. 2.1 Word Embedding (Word Representation in ML) 67 Dense Representation (Improved Word2Vec) "apple" : "ap", "app", "appl", "apple"
- 68. 2.1 Word Embedding (Word Representation in ML) 68 Word Embedding with Pre-training
- 69. 2.1 Word Embedding (Word Representation in ML) 69 But They can not consider contextual meaning of word
- 70. 2.1 Word Embedding (Word Representation in ML) 70 I Love an Apple. It is delicious than Banana. I Love the Apple. It is better than Samsung. same embedding vector… is it ok?
- 71. 2.1 Word Embedding (Word Representation in ML) 71 School, Home, Hospital, Church, Temple … They will have similar embedding vector. Because they only consider about whether they appear at the same time [going, to , _____ , I , am] are appear at the same time. 4/28 : Today, I am going to _____ , I am so happy. 4/29 : Today, I am going to _____ , I am so sad… 4/30 : Today, I am going to _____ , I am so exciting!
- 72. 2.1 Word Embedding (Word Representation in ML) 72 But, think again... Is Church similar with Temple? Is School similar with Home? …Really? ~~ ~~
- 73. 73 2.2 Embedding from Language Model (ELMo)
- 74. 2.2 Embedding from Language Model (ELMo) 74 Language Model : To assign probability to sequence. Consider previous words and Predict next word
- 75. 2.2 Embedding from Language Model (ELMo) 75 Consider neighbor word Word2Vec Consider previous words Language Model Language Model : To assign probability to sequence.
- 76. 2.2 Embedding from Language Model (ELMo) 76 Embedding from Language Model (ELMo) ELMo uses 2 Language Model consisted of Multi Layer RNN(LSTM) : BiLM
- 77. 2.2 Embedding from Language Model (ELMo) 77 Embedding from Language Model (ELMo) To embed the word ”play”, ELMo uses the result of each layer inside the dotted rectangle above as the material.
- 78. 2.2 Embedding from Language Model (ELMo) 78 Embedding from Language Model (ELMo) 1. Concatenate Forward LM & Backward LM 2. Multiply Wight at each layer’s outputs 3. Add Every layer’s outputs 4. Scale by multiplying constant
- 79. 2.2 Embedding from Language Model (ELMo) 79 Embedding from Language Model (ELMo) We can use ELMo representation for downstream task with embedding And Now, Model knows about word’s contextual meaning.
- 80. 2.2 Embedding from Language Model (ELMo) 80 Embedding from Language Model (ELMo) But, there are still a big problem We still lack the data specific to a particular task
- 81. 2.2 Embedding from Language Model (ELMo) 81 We still lack the data specific to a particular task Text Classification
- 82. 2.2 Embedding from Language Model (ELMo) 82 We still lack the data specific to a particular task Named Entity Recognition
- 83. 2.2 Embedding from Language Model (ELMo) 83 We still lack the data specific to a particular task Machine Translation
- 84. 2.2 Embedding from Language Model (ELMo) 84 We still lack the data specific to a particular task Question & Answering
- 85. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 85
- 86. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 86 But we have a lot of unlabeled text data.
- 87. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 87 But we have a lot of unlabeled text data.
- 88. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 88 We want universal NLU Model trained unlabeled data like human So, GPT tries semi-supervised approach with pre-train & fine tune to solve that problem (lack of task specific dataset)
- 89. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 89 Unsupervised Pre-train Unsupervised Corpus of token U = u1, u2, … , un The maximum likelihood of maximizing a standard LM objective is: K = window size Theta = NN’s parameter
- 90. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 90 Unsupervised Pre-train Unsupervised Corpus of token U = u1, u2, … , un We = Embedding , Wp = Positional P(u) is output of model (next word prediction)
- 91. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 91 Unsupervised Pre-train
- 92. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 92 Supervised Fine Tune
- 93. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 93 Supervised Fine Tune Supervised Corpus of token X = x1, x2, … , xm, label = Y pass through the linear output layer once more to predict the y value
- 94. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 94 Auxiliary Loss Supervised Corpus of token X = x1, x2, … , xm, label = Y pass through the linear output layer once more to predict the y value
- 95. 2.3 Unsupervised Pre-train, Supervised Fine Tune (GPT) 95 Framework
- 96. 96 3. BERT & Experiments
- 97. 97 3. BERT & Experiments Pre-training Reference : Kim Dong Hwa
- 98. 98 3. BERT & Experiments Pre-training Reference : Kim Dong Hwa
- 99. 99 3. BERT & Experiments Pre-training Reference : Kim Dong Hwa
- 100. 100 3. BERT & Experiments Pre-training : Segmentation Reference : Kim Dong Hwa
- 101. 101 3. BERT & Experiments Pre-training : Segmentation Reference : Kim Dong Hwa
- 102. 102 3. BERT & Experiments Pre-training : Segmentation Reference : Kim Dong Hwa
- 103. 103 3. BERT & Experiments Pre-training : Segmentation Reference : Kim Dong Hwa
- 104. 104 3. BERT & Experiments Pre-training : Segmentation Reference : Kim Dong Hwa
- 105. 105 3. BERT & Experiments Preprocessing : Next Sentence Reference : Kim Dong Hwa
- 106. 106 3. BERT & Experiments Preprocessing : Next Sentence Reference : Kim Dong Hwa
- 107. 107 3. BERT & Experiments Preprocessing : Length & Mask Reference : Kim Dong Hwa
- 108. 108 3. BERT & Experiments Preprocessing : Length & Mask Reference : Kim Dong Hwa
- 109. 109 3. BERT & Experiments Preprocessing : 학습되는 데이터 Reference : Kim Dong Hwa
- 110. 110 3. BERT & Experiments BERT vs Transformer Reference : Kim Dong Hwa
- 111. 111 3. BERT & Experiments BERT vs Transformer Reference : Kim Dong Hwa
- 112. 112 3. BERT & Experiments BERT vs Transformer Reference : Kim Dong Hwa
- 113. 113 3. BERT & Experiments BERT vs Transformer Reference : Kim Dong Hwa
- 114. 114 3. BERT & Experiments BERT vs Transformer Reference : Kim Dong Hwa
- 115. 115 3. BERT & Experiments Token Embedding Reference : Kim Dong Hwa
- 116. 116 3. BERT & Experiments Segment Embedding Reference : Kim Dong Hwa
- 117. 117 3. BERT & Experiments Positional Embedding Reference : Kim Dong Hwa
- 118. 118 3. BERT & Experiments Norm Reference : Kim Dong Hwa
- 119. 119 3. BERT & Experiments GELU Reference : Kim Dong Hwa
- 120. 120 3. BERT & Experiments Forwarding Reference : Kim Dong Hwa
- 121. 121 3. BERT & Experiments Loss Reference : Kim Dong Hwa
- 122. 122 3. BERT & Experiments Next Sentence Loss Reference : Kim Dong Hwa
- 123. 123 3. BERT & Experiments Next Sentence Loss Reference : Kim Dong Hwa
- 124. 124 3. BERT & Experiments Mask Loss Reference : Kim Dong Hwa
- 125. 125 3. BERT & Experiments Mask Loss Reference : Kim Dong Hwa
- 126. 126 3. BERT & Experiments Fine Tune Reference : Kim Dong Hwa
- 127. 127 3. BERT & Experiments Fine Tune Reference : Kim Dong Hwa
- 128. 128 3. BERT & Experiments Experiment Result Reference : Kim Dong Hwa
- 129. 129 3. BERT & Experiments Experiment Result Reference : Kim Dong Hwa