TensorFlow 深度學習快速上手班--深度學習
•Download as PPTX, PDF•
24 likes•5,293 views
http://www.hcinnovation.tw/tc/index.php/k2-blog/item/66-tensorflow 課程講義2
Recommended
More Related Content
What's hot
What's hot (20)
Similar to TensorFlow 深度學習快速上手班--深度學習
Similar to TensorFlow 深度學習快速上手班--深度學習 (20)
More from Mark Chang
More from Mark Chang (20)
TensorFlow 深度學習快速上手班--深度學習
- 2. • 深度學習的原理 • 模型選擇與參數調整 • 多層感知器實作
- 3. 深度學習的原理
- 4. 機器學習 監督式學習 Supervised Learning 非監督式學習 Unsupervised Learning 增強式學習 Reinforcement Learning 深度學習 Deep Learning
- 5. 深度學習 • 一種機器學習的方法 • 用電腦模擬人腦神經系統構造 • 讓電腦學會人腦可做的事
- 9. 二元分類：AND Gate x1 x2 y 0 0 0 0 1 0 1 0 0 1 1 1 (0,0) (0,1) (1,1) (1,0) 0 1 n20 20 b -30 yx 1 x 2
- 10. XOR Gate ? (0,0) (0,1) (1,1) (1,0) 0 0 1 x1 x2 y 0 0 0 0 1 1 1 0 1 1 1 0
- 11. 二元分類：XOR Gate n -20 20 b -10 y (0,0) (0,1) (1,1) (1,0) 0 1 (0,0) (0,1) (1,1) (1,0) 1 0 (0,0) (0,1) (1,1) (1,0) 0 0 1 n1 20 20 b -30 x 1 x 2 n2 20 20 b -10 x 1 x 2 x1 x2 n1 n2 y 0 0 0 0 0 0 1 0 1 1 1 0 0 1 1 1 1 1 1 0
- 14. 訓練類神經網路 • 用隨機值初始化模型參數w • Forward Propagation – 用目前的模型參數計算出答案 • 計算錯誤量（用Error Function） • Backward Propagation – 用錯誤量來修正模型
- 16. 訓練類神經網路 訓練資料 機器學習模型 輸出值 正確答案 對答案 如果答錯了， 要修正模型 初始化 Forward Propagatio n Error Function Backward Propagatio n
- 17. 初始化 • 將所有的W隨機設成-N～N之間的數 • 每層之間W的值都不能相同 x y n11 n12 n21 n22W12,y W12,x b W11,y W11,bW12,b b W11,x W21,11 W22,12 W21,12 W22,11 W21,bW22,b z 1 z 2 Lk-1：上一層的大小 Lk ：該層的大小
- 30. 模型選擇與參數調整
- 32. 模型種類 • Hidden Layer 較小的Hidden Layer 較大的Hidden Layer 多層Hidden Layer單層Hidden Layer
- 35. 訓練不足（Underfitting） • 原因： – Learning Rate 太大或太 小 – 訓練時間太短 – 模型複雜度不夠 t
- 36. 過度訓練（Overfitting） • 原因： – 雜訊太多 – 訓練資料太少 – 訓練時間太長 – 模型複雜度太高 t
- 37. 驗證資料（Validation Data） 訓練資料 模型 1 測試資料 最後結果 資料集 驗證資料 模型選擇 參數選擇 時間控制 模型 2 ……
- 39. 解決方式 • 訓練不足 – 調整Learning Rate – 增加訓練時間 – 增加模型複雜度 • 訓練過度 – 增加訓練資料 – 減少雜訊 – 減少訓練時間 – 減少模型複雜度
- 40. 調整Learning Rate • 調整Learning Rate數值 Learning Rate 適中 Learning Rate 過小 Learning Rate 過大
- 41. 調整Learning Rate • 動態調整Learning Rate： – AdagradOptimizer – RMSPropOptimizer – ……
- 42. 調整訓練時間 • Early Stop Validation LossTraining Loss 停止訓練 t
- 43. 調整模型複雜度 • 調整Hidden Layer的寬度或層數 • Regularization • Dropout
- 44. Hidden Layer寬度 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 1 2 3 4 5 6 7 8 9 Validation Loss Training Loss 最適寬度 模型複雜度低 高 Loss 寬度
- 45. Regularization • 將weights的平方和加到cost function中 • 可使weights的絕對值不要變得太大 • 可降低模型複雜度 Cost Function: λ越大，則模型複雜度越低
- 46. Regularization 最適λ值 模型複雜度 低高 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.01 0.1 1 Validation Loss Training Loss Loss λ
- 47. Dropout • 訓練時，隨機將Hidden Layer的神經元拿掉 • 可降低模型複雜度 • ex: 25%的Dropout Rate
- 48. Dropout • 測試時，用所有的神經元來測試。 – 將所有的weight乘上 (1 – dropout_rate)
- 49. Dropout 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 0.2 0.4 0.6 0.8 1 Validation Error Training Error 最適dropout rate 1- dropout rate Error 模型複雜度低 高
- 50. 模型選擇與參數調整實作 • Tensorflow Playground – http://playground.tensorflow.org/
- 53. 多層感知器實作
- 56. 模型 • 多層感知器 Input Layer Size:784 Hidden Layer Size:200 Output Layer Size:10
- 57. Computational Graph x_ = tf.placeholder(tf.float32, [None, 784], name="x_") y_ = tf.placeholder(tf.float32, [None, 10], name="y_") # input -> Hidden W1 = tf.Variable(tf.truncated_normal([784,200], stddev=0.1), name="W1") b1 = tf.Variable(tf.zeros([200]), name="b1") h1 = tf.nn.sigmoid(tf.matmul(x_, W1) + b1) # Hidden -> Output W2 = tf.Variable(tf.truncated_normal([200,10], stddev=0.1), name="W2") b2 = tf.Variable(tf.zeros([10]), name="b2") y = tf.nn.softmax(tf.matmul(h1, W2) + b2) cross_entropy = -tf.reduce_sum(y_ * tf.log(y)) optimizer = tf.train.GradientDescentOptimizer(0.01) trainer = optimizer.minimize(cross_entropy) init = tf.initialize_all_variables()
- 58. Layer 1 W1 = tf.Variable(tf.truncated_normal([784,200], stddev=0.1), name="W1”) 0 1000 2000 3000 4000 5000 6000 7000 -0.2 0.20
- 59. Layer 1 W1 = tf.Variable(tf.truncated_normal([784,200], stddev=0.1), name="W1") b1 = tf.Variable(tf.zeros([200]), name="b1") h1 = tf.nn.sigmoid(tf.matmul(x_, W1) + b1) W1 x b1 h1 n 784 n 200 200 200 784 × + =
- 60. Layer 2 w b h1 n 10 10 200 200× + = y 10 n W2 = tf.Variable(tf.truncated_normal([200,10], stddev=0.1), name="W2") b2 = tf.Variable(tf.zeros([10]), name="b2") y = tf.nn.softmax(tf.matmul(h1, W2) + b2)
- 61. Regularization lambda_ = tf.placeholder(tf.float32, name="lambda") regularizer = tf.reduce_sum(tf.square(W1)) +tf.reduce_sum(tf.square(W2)) cost = cross_entropy + lambda_*regularizer Cost Function:
- 63. dropout keep_prob = tf.placeholder(tf.float32, name="keep_prob") h1_drop = tf.nn.dropout(h1, keep_prob) y = tf.nn.softmax(tf.matmul(h1_drop, W2) + b2) 1 0 1 0 Dropout Mask
- 65. 模型儲存與載入 • 儲存模型參數 • 載入模型參數 saver = tf.train.Saver(max_to_keep=10) saver.save(sess, "model.ckpt") saver = tf.train.Saver() saver.restore(sess, "model.ckpt")
- 66. 講師資訊 • Email: ckmarkoh at gmail dot com • Blog: http://cpmarkchang.logdown.com • Github: https://github.com/ckmarkoh Mark Chang • Facebook: https://www.facebook.com/ckmarkoh.chang • Slideshare: http://www.slideshare.net/ckmarkohchang • Linkedin: https://www.linkedin.com/pub/mark- chang/85/25b/847 66
Editor's Notes
- y = \frac{1}{ 1+e^{- ( w_{1} x_{1} + w_{2}x_{2}+w_{b} ) }} & n_{in} = w_{1} x_{1} + w_{2}x_{2}+w_{b} \\ & n_{out} = \frac{1}{1+e^{-n_{in}}}
- w_{1}x_{1}+w_{2}x_{2}+w_{b} = 0 w_{1}x_{1}+w_{2}x_{2}+w_{b} < 0 w_{1}x_{1}+w_{2}x_{2}+w_{b} >0
- y = \frac{1}{1+e^{-(20x_{1}+20x_{2}-30)}} 20x_{1}+20x_{2}-30 = 0
- & J = -( z_{1} log(n_{21(out)}) + (1-z_{1}) log (1 -n_{21(out)} )) \\ &\mspace{30mu} -( z_{2} log(n_{22(out)}) + (1-z_{2}) log (1 -n_{22(out)} )) \\ & n_{out} \approx 0 \text{ and } z = 0 \Rightarrow J \approx 0 \\ & n_{out} \approx 1 \text{ and } z = 1 \Rightarrow J \approx 0 \\ & n_{out} \approx 0 \text{ and } z = 1 \Rightarrow J \approx \infty \\ & n_{out} \approx 1 \text{ and } z = 0 \Rightarrow J \approx \infty \\
- & w_{21,11} \leftarrow w_{21,11} - \eta \dfrac{\partial J}{\partial w_{21,11}} \\ & w_{21,12} \leftarrow w_{21,12} - \eta \dfrac{\partial J}{\partial w_{21,12}} \\ & w_{21,b} \leftarrow w_{21,b} - \eta \dfrac{\partial J}{\partial w_{21,b}} \\ & w_{22,11} \leftarrow w_{21,11} - \eta \dfrac{\partial J}{\partial w_{22,11}} \\ & w_{22,12} \leftarrow w_{21,12} - \eta \dfrac{\partial J}{\partial w_{22,12}} \\ & w_{22,b} \leftarrow w_{21,b} - \eta \dfrac{\partial J}{\partial w_{22,b}} \\ &w_{11,x} \leftarrow w_{11,x} - \eta \dfrac{\partial J}{\partial w_{11,x}} \\ &w_{11,y} \leftarrow w_{11,y} - \eta \dfrac{\partial J}{\partial w_{11,y}} \\ &w_{11,b} \leftarrow w_{11,b} - \eta \dfrac{\partial J}{\partial w_{11,b}} \\ &w_{12,x} \leftarrow w_{12,x} - \eta \dfrac{\partial J}{\partial w_{12,x}} \\ &w_{12,y} \leftarrow w_{12,y} - \eta \dfrac{\partial J}{\partial w_{12,y}} \\ &w_{12,b} \leftarrow w_{12,b} - \eta \dfrac{\partial J}{\partial w_{12,b}} \\ ( – \dfrac{ \partial J}{\partial w_{0}} , – \dfrac{ \partial J}{\partial w_{1}} )
- \dfrac{\partial J}{\partial w_{21,11}} = \dfrac{\partial J}{\partial n_{21(out)}} \dfrac{\partial n_{21(out)}}{\partial n_{21(in)}} \dfrac{\partial n_{21(in)}}{\partial w_{21,11}} = (n_{21(out)}-z_{1}) n_{11(out)} \\ \delta_{21(out)} \delta_{21(in)} n_{11(out)} w_{21,11} \leftarrow w_{21,11} - \eta
- \dfrac{\partial J}{\partial w_{11,x}} = \dfrac{\partial J}{\partial n_{21(out)}} \dfrac{\partial n_{21(out)}}{\partial n_{21(in)}} \dfrac{\partial n_{21(in)}}{\partial w_{21,11}} w_{11,x} \leftarrow w_{11,x} - \eta \delta_{11(in)} x
- & {\color[rgb]{0.597455,0.000000,0.759310}\delta_{11(in)}} =\dfrac{\partial J}{\partial n_{11(in)}} ={\color[rgb]{1.000000,0.500000,0.000000}\dfrac{\partial J}{\partial n_{21(out)}} } \dfrac{\partial n_{21(out)}}{\partial n_{11(in)}} + {\color[rgb]{1.000000,0.500000,0.000000}\dfrac{\partial J}{\partial n_{22(out)}}} \dfrac{\partial n_{22(out)}}{\partial n_{11(in)}} \\ & {\color[rgb]{0.597455,0.000000,0.759310}\delta_{11(in)}} =\dfrac{\partial J}{\partial n_{11(in)}} ={\color[rgb]{1.000000,0.500000,0.000000}\dfrac{\partial J}{\partial n_{21(out)}} } \dfrac{\partial n_{21(out)}}{\partial n_{11(in)}} + {\color[rgb]{1.000000,0.500000,0.000000}\dfrac{\partial J}{\partial n_{22(out)}}} \dfrac{\partial n_{22(out)}}{\partial n_{11(in)}} \\ &= {\color[rgb]{1.000000,0.500000,0.000000}\dfrac{\partial J}{\partial n_{21(out)}}} {\color[rgb]{1.000000,0.000000,0.000000}\dfrac{\partial n_{21(out)}}{\partial n_{21(in)}} } {\color[rgb]{0.795165,0.000000,0.447221}\dfrac{\partial n_{21(in)}}{\partial n_{11(out)}} } {\color[rgb]{0.597455,0.000000,0.759310}\dfrac{\partial n_{11(out)}}{\partial n_{11(in)}} } + {\color[rgb]{1.000000,0.500000,0.000000}\dfrac{\partial J_{2}}{\partial n_{22(out)}} } {\color[rgb]{1.000000,0.000000,0.000000}\dfrac{\partial n_{22(out)}}{\partial n_{22(in)}} } {\color[rgb]{0.795165,0.000000,0.447221}\dfrac{\partial n_{22(in)}}{\partial n_{11(out)}} } {\color[rgb]{0.597455,0.000000,0.759310}\dfrac{\partial n_{11(out)}}{\partial n_{11(in)}}} \\ &= ({\color[rgb]{1.000000,0.500000,0.000000}\dfrac{\partial J}{\partial n_{21(out)}}} {\color[rgb]{1.000000,0.000000,0.000000}\dfrac{\partial n_{21(out)}}{\partial n_{21(in)}} } {\color[rgb]{0.795165,0.000000,0.447221}\dfrac{\partial n_{21(in)}}{\partial n_{11(out)}} } + {\color[rgb]{1.000000,0.500000,0.000000}\dfrac{\partial J_{2}}{\partial n_{22(out)}} } {\color[rgb]{1.000000,0.000000,0.000000}\dfrac{\partial n_{22(out)}}{\partial n_{22(in)}} } {\color[rgb]{0.795165,0.000000,0.447221}\dfrac{\partial n_{22(in)}}{\partial n_{11(out)}} }) {\color[rgb]{0.597455,0.000000,0.759310}\dfrac{\partial n_{11(out)}}{\partial n_{11(in)}}} \\ &= ( {\color[rgb]{1.000000,0.000000,0.000000}\delta_{21(in)} } {\color[rgb]{0.795165,0.000000,0.447221}w_{21,11} } + {\color[rgb]{1.000000,0.000000,0.000000}\delta_{22(in)} } {\color[rgb]{0.795165,0.000000,0.447221}w_{22,11} }) {\color[rgb]{0.597455,0.000000,0.759310}\dfrac{\partial n_{11(out)}}{\partial n_{11(in)}}} \\
- J = cross\_entropy + \lambda \sum_{i,j} w_{i,j}^{2}
- & x_{1}^{(2)}\\ & x_{2}^{(2)} & x_{1}^{(1)}\\ & x_{2}^{(1)} y^{(1)} y^{(2)}
- w \leftarrow w(1-dropout\_rate)