Successfully reported this slideshow.

# Backpropagation: Understanding How to Update ANNs Weights Step-by-Step

This presentation explains how the backpropagation algorithm is useful in updating the artificial neural networks (ANNs) weights using two examples step by step. Readers should have a basic understanding of how ANNs work, partial derivatives, and multivariate chain rule.

This presentation won`t dive directly into the details of the algorithm but will start by training a very simple network. This is because the backpropagation algorithm is meant to be applied over a network after training. So, we should train the network before applying it to catch the benefits of backpropagation algorithm and how to use it.

• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

### Backpropagation: Understanding How to Update ANNs Weights Step-by-Step

1. 1. Backpropagation: Understanding How to Update ANNs Weights Step-by-Step Ahmed Fawzy Gad ahmed.fawzy@ci.menofia.edu.eg MENOUFIA UNIVERSITY FACULTY OF COMPUTERS AND INFORMATION INFORMATION TECHNOLOGY โซุงููููููุฉโฌ โซุฌุงูุนุฉโฌ โซูุงููุนูููุงุชโฌ โซุงูุญุงุณุจุงุชโฌ โซูููุฉโฌ โซุงููุนูููุงุชโฌ โซุชูููููุฌูุงโฌ โซุงููููููุฉโฌ โซุฌุงูุนุฉโฌ
2. 2. Train then Update โข The backpropagation algorithm is used to update the NN weights when they are not able to make the correct predictions. Hence, we should train the NN before applying backpropagation. Initial Weights PredictionTraining
3. 3. Train then Update โข The backpropagation algorithm is used to update the NN weights when they are not able to make the correct predictions. Hence, we should train the NN before applying backpropagation. Initial Weights PredictionTraining BackpropagationUpdate
4. 4. Neural Network Training Example ๐ ๐ ๐ ๐ ๐๐ฎ๐ญ๐ฉ๐ฎ๐ญ ๐. ๐ ๐. ๐ ๐. ๐๐ ๐๐ ๐๐ ๐ ๐. ๐ ๐. ๐ 1. ๐๐ Training Data Initial Weights ๐. ๐ In Out ๐พ ๐ = ๐. ๐ ๐พ ๐ = ๐. ๐ +๐ ๐ = ๐. ๐๐ ๐. ๐ ๐ฟ ๐ In Out ๐พ ๐ ๐พ ๐ +๐ ๐ ๐ฟ ๐
5. 5. Network Training โข Steps to train our network: 1. Prepare activation function input (sum of products between inputs and weights). 2. Activation function output. ๐. ๐ In Out ๐พ ๐ = ๐. ๐ ๐พ ๐ = ๐. ๐ +๐ ๐ = ๐. ๐๐ ๐. ๐
6. 6. Network Training: Sum of Products โข After calculating the sop between inputs and weights, next is to use this sop as the input to the activation function. ๐. ๐ In Out ๐พ ๐ = ๐. ๐ ๐พ ๐ = ๐. ๐ +๐ ๐ = ๐. ๐๐ ๐. ๐ ๐ = ๐ฟ1 โ ๐พ1 + ๐ฟ2 โ ๐พ2 + ๐ ๐ = ๐. ๐ โ ๐. ๐ + ๐. ๐ โ ๐. ๐ + ๐. ๐๐ ๐ = ๐. ๐๐
7. 7. Network Training: Activation Function โข In this example, the sigmoid activation function is used. โข Based on the sop calculated previously, the output is as follows: ๐. ๐ In Out ๐พ ๐ = ๐. ๐ ๐พ ๐ = ๐. ๐ +๐ ๐ = ๐. ๐๐ ๐. ๐ ๐ ๐ = ๐ ๐ + ๐โ๐ ๐ ๐ = ๐ ๐ + ๐โ๐.๐๐ = ๐ ๐ + ๐. ๐๐๐ = ๐ ๐. ๐๐๐ ๐ ๐ = ๐. ๐๐๐
8. 8. Network Training: Prediction Error โข After getting the predicted outputs, next is to measure the prediction error of the network. โข We can use the squared error function defined as follows: โข Based on the predicted output, the prediction error is: ๐. ๐ In Out ๐พ ๐ = ๐. ๐ ๐พ ๐ = ๐. ๐ +๐ ๐ = ๐. ๐๐ ๐. ๐ ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ ๐ฌ = ๐ ๐ ๐. ๐๐ โ ๐. ๐๐๐ ๐ = ๐ ๐ โ๐. ๐๐๐ ๐ = ๐ ๐ ๐. ๐๐๐ = ๐. ๐๐๐
9. 9. How to Minimize Prediction Error? โข There is a prediction error and it should be minimized until reaching an acceptable error. What should we do in order to minimize the error? โข There must be something to change in order to minimize the error. In our example, the only parameter to change is the weight. How to update the weights? โข We can use the weights update equation: ๐พ ๐๐๐ = ๐พ ๐๐๐ + ฮท ๐ โ ๐ ๐ฟ
10. 10. Weights Update Equation โข We can use the weights update equation: ๏ง ๐พ ๐๐๐: new updated weights. ๏ง ๐พ ๐๐๐: current weights. [1.83, 0.5, 0.2] ๏ง ฮท: network learning rate. 0.01 ๏ง ๐: desired output. 0.03 ๏ง ๐: predicted output. 0.874 ๏ง ๐ฟ: current input at which the network made false prediction. [+1, 0.1, 0.3] ๐พ ๐๐๐ = ๐พ ๐๐๐ + ฮท ๐ โ ๐ ๐ฟ
11. 11. Weights Update Equation ๐พ ๐๐๐ = ๐พ ๐๐๐ + ฮท ๐ โ ๐ ๐ฟ = [๐. ๐๐, ๐. ๐, ๐. ๐ + ๐. ๐๐ ๐. ๐๐ โ ๐. ๐๐๐ [+๐, ๐. ๐, ๐. ๐ = [๐. ๐๐, ๐. ๐, ๐. ๐ + โ๐. ๐๐๐๐[+๐, ๐. ๐, ๐. ๐ = [๐. ๐๐, ๐. ๐, ๐. ๐ + [โ๐. ๐๐๐๐, โ๐. ๐๐๐๐๐, โ๐. ๐๐๐๐ = [๐. ๐๐๐, ๐. ๐๐๐, ๐. ๐๐๐
12. 12. Weights Update Equation โข The new weights are: โข Based on the new weights, the network will be re-trained. ๐พ ๐๐๐๐ ๐พ ๐๐๐๐ ๐ ๐๐๐ ๐. ๐๐๐ ๐. ๐๐๐ ๐. ๐๐๐ ๐. ๐ In Out ๐พ ๐ = ๐. ๐ ๐พ ๐ = ๐. ๐ +๐ ๐ = ๐. ๐๐ ๐. ๐
13. 13. Weights Update Equation โข The new weights are: โข Based on the new weights, the network will be re-trained. โข Continue these operations until prediction error reaches an acceptable value. 1. Updating weights. 2. Retraining network. 3. Calculating prediction error. ๐พ ๐๐๐๐ ๐พ ๐๐๐๐ ๐ ๐๐๐ ๐. ๐๐๐ ๐. ๐๐๐ ๐. ๐๐๐ ๐. ๐ In Out ๐พ ๐ = ๐. ๐๐๐ ๐พ ๐ = ๐. ๐๐๐ +๐ ๐ = ๐. ๐22 ๐. ๐
14. 14. Why Backpropagation Algorithm is Important? โข The backpropagation algorithm is used to answer these questions and understand effect of each weight over the prediction error. New Weights !Old Weights
15. 15. Forward Vs. Backward Passes โข When training a neural network, there are two passes: forward and backward. โข The goal of the backward pass is to know how each weight affects the total error. In other words, how changing the weights changes the prediction error? Forward Backward
16. 16. Backward Pass โข Let us work with a simpler example: โข How to answer this question: What is the effect on the output Y given a change in variable X? โข This question is answered using derivatives. Derivative of Y wrt X ( ๐๐ ๐๐ฟ ) will tell us the effect of changing the variable X over the output Y. ๐ = ๐ฟ ๐ ๐ + ๐ฏ
17. 17. Calculating Derivatives โข The derivative ๐๐ ๐๐ฟ can be calculated as follows: โข Based on these two derivative rules: โข The result will be: ๐๐ ๐๐ฟ = ๐ ๐๐ฟ (๐ฟ ๐ ๐ + ๐ฏ) ๐ = ๐ฟ ๐ ๐ + ๐ฏ ๐ ๐๐ฟ ๐ฟ ๐ = ๐๐ฟSquare ๐ ๐๐ฟ ๐ช = ๐Constant ๐๐ ๐๐ฟ = ๐๐ฟ๐ + ๐ = ๐๐ฟ๐
18. 18. Prediction Error โ Weight Derivative E W? ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ Change in Y wrt X ๐๐ ๐๐ฟ Change in E wrt W ๐๐ฌ ๐๐พ
19. 19. Prediction Error โ Weight Derivative ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐
20. 20. Prediction Error โ Weight Derivative ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐
21. 21. Prediction Error โ Weight Derivative ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ ๐๐๐๐๐๐๐ = ๐. ๐๐ (๐ช๐๐๐๐๐๐๐)
22. 22. Prediction Error โ Weight Derivative ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ ๐๐๐๐๐๐๐ = ๐. ๐๐ (๐ช๐๐๐๐๐๐๐) ๐ท๐๐๐๐๐๐๐๐ = ๐ ๐ = ๐ ๐ + ๐โ๐
23. 23. Prediction Error โ Weight Derivative ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐ ๐ + ๐โ๐ ๐ ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ ๐๐๐๐๐๐๐ = ๐. ๐๐ (๐ช๐๐๐๐๐๐๐) ๐ท๐๐๐๐๐๐๐๐ = ๐ ๐ = ๐ ๐ + ๐โ๐
24. 24. Prediction Error โ Weight Derivative ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐ ๐ + ๐โ๐ ๐ ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ ๐๐๐๐๐๐๐ = ๐. ๐๐ (๐ช๐๐๐๐๐๐๐) ๐ท๐๐๐๐๐๐๐๐ = ๐ ๐ = ๐ ๐ + ๐โ๐
25. 25. Prediction Error โ Weight Derivative ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐ ๐ + ๐โ๐ ๐ ๐ = ๐ฟ1 โ ๐พ1 + ๐ฟ2 โ ๐พ2 + ๐ ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ ๐๐๐๐๐๐๐ = ๐. ๐๐ (๐ช๐๐๐๐๐๐๐) ๐ท๐๐๐๐๐๐๐๐ = ๐ ๐ = ๐ ๐ + ๐โ๐
26. 26. Prediction Error โ Weight Derivative ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐ ๐ + ๐โ๐ ๐ ๐๐๐๐๐๐๐ = ๐. ๐๐ (๐ช๐๐๐๐๐๐๐) ๐ท๐๐๐๐๐๐๐๐ = ๐ ๐ = ๐ ๐ + ๐โ๐ ๐ = ๐ฟ1 โ ๐พ1 + ๐ฟ2 โ ๐พ2 + ๐ ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐ ๐ + ๐โ(๐ฟ1โ ๐พ1+ ๐ฟ2โ๐พ2+๐) ๐
27. 27. Multivariate Chain Rule Predicted Output Prediction Error sop Weights ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ ๐ ๐ = ๐ ๐ + ๐โ๐ ๐ = ๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐พ ๐, ๐พ ๐ ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐ ๐ + ๐โ(๐ฟ1โ ๐พ1+ ๐ฟ2โ๐พ2+๐) ๐ ๐๐ฌ ๐๐พ = ๐ ๐๐พ ( ๐ ๐ ๐๐๐๐๐๐๐ โ ๐ ๐ + ๐โ(๐ฟ ๐โ ๐พ ๐+ ๐ฟ ๐โ๐พ ๐+๐) ๐ ) Chain Rule
28. 28. Multivariate Chain Rule Predicted Output Prediction Error sop Weights ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐ ๐ ๐ = ๐ ๐ + ๐โ๐ ๐ = ๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐พ ๐, ๐พ ๐ ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ ๐๐ ๐๐พ ๐ ๐๐ ๐๐พ ๐ ๐๐ฌ ๐๐พ ๐ ๐๐ฌ ๐๐พ ๐ Letโs calculate these individual partial derivatives. ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ โ ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ โ ๐๐ ๐๐พ ๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ โ ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ โ ๐๐ ๐๐พ ๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ โ ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ โ ๐๐ ๐๐พ ๐
29. 29. Error-Predicted ( ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ ) Partial Derivative Substitution ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ = ๐ ๐๐ท๐๐๐๐๐๐๐๐ ( ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐) = ๐ โ ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐โ๐ โ (๐ โ ๐) )= (๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐) โ (โ๐ = ๐๐๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐ ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ = ๐๐๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐ = ๐. ๐๐๐ โ ๐. ๐๐ ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ = ๐. ๐๐๐ ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐๐๐๐๐๐๐ ๐
30. 30. Predicted-sop ( ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ ) Partial Derivative ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ = ๐ ๐๐ ( ๐ ๐ + ๐โ๐ ) ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ = ๐ ๐ + ๐โ๐ (๐ โ ๐ ๐ + ๐โ๐ ) ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ = ๐ ๐ + ๐โ๐ (๐ โ ๐ ๐ + ๐โ๐ ) = ๐ ๐ + ๐โ๐.๐๐ (๐ โ ๐ ๐ + ๐โ๐.๐๐ ) = ๐ ๐ + ๐. ๐๐๐ (๐ โ ๐ ๐ + ๐. ๐๐๐ ) = ๐ ๐. ๐๐๐ (๐ โ ๐ ๐. ๐๐๐ ) = ๐. ๐๐๐(๐ โ ๐. ๐๐๐) = ๐. ๐๐๐(๐. ๐๐๐) ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ = ๐. ๐๐ Substitution ๐๐ซ๐๐๐ข๐๐ญ๐๐ = ๐ ๐ + ๐โ๐
31. 31. Sop-๐1 ( ๐๐ ๐๐พ ๐ ) Partial Derivative ๐๐ ๐๐พ ๐ = ๐ ๐๐พ ๐ (๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐) = ๐ โ ๐ฟ ๐ โ ๐พ ๐ ๐โ๐ + ๐ + ๐ = ๐ฟ ๐ โ ๐พ ๐ ๐ )= ๐ฟ ๐(๐ ๐๐ ๐๐พ ๐ = ๐ฟ ๐ ๐๐ ๐๐พ ๐ = ๐ฟ ๐ Substitution ๐๐ ๐๐พ ๐ = ๐. ๐ ๐ฌ = ๐ฟ1 โ ๐พ1 + ๐ฟ2 โ ๐พ2 + ๐
32. 32. ๐๐ ๐๐พ ๐ = ๐ ๐๐พ ๐ (๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐) = ๐ + ๐ โ ๐ฟ ๐ โ ๐พ ๐ ๐โ๐ + ๐ = ๐ฟ ๐ โ ๐พ ๐ ๐ )= ๐ฟ ๐(๐ ๐๐ ๐๐พ ๐ = ๐ฟ ๐ ๐๐ ๐๐พ ๐ = ๐ฟ ๐ = ๐. ๐ Substitution ๐๐ ๐๐พ ๐ = ๐. ๐ ๐ฌ = ๐ฟ1 โ ๐พ1 + ๐ฟ2 โ ๐พ2 + ๐ Sop-๐1 ( ๐๐ ๐๐พ ๐ ) Partial Derivative
33. 33. Error-๐1 ( ๐๐ฌ ๐๐พ ๐ ) Partial Derivative โข After calculating each individual derivative, we can multiply all of them to get the desired relationship between the prediction error and each weight. ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ โ ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ โ ๐๐ ๐๐พ ๐ ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ = ๐. ๐๐๐ ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ = ๐. ๐๐ ๐๐ ๐๐พ ๐ = ๐. ๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ โ ๐. ๐๐ โ ๐. ๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐ Calculated Derivatives
34. 34. Error-๐2 ( ๐๐ฌ ๐๐พ ๐ ) Partial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ โ ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ โ ๐๐ ๐๐พ ๐ ๐๐ฌ ๐๐ท๐๐๐๐๐๐๐๐ = ๐. ๐๐๐ ๐๐ท๐๐๐๐๐๐๐๐ ๐๐ = ๐. ๐๐ ๐๐ ๐๐พ ๐ = ๐. ๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ โ ๐. ๐๐ โ ๐. ๐ Calculated Derivatives
35. 35. Interpreting Derivatives โข There are two useful pieces of information from the derivatives calculated previously. Increasing/decreasing weight increases/decreases error. Derivative MagnitudeDerivative Sign Positive Increasing/decreasing weight decreases/increases error. Negative Increasing/decreasing weight by P increases/decreases error by MAG*P. Increasing/decreasing weight by P decreases/increases error by MAG*P. Positive Sign Negative Sign In our example, because both ๐๐ฌ ๐๐พ ๐ and ๐๐ฌ ๐๐พ ๐ are positive, then we would like to decrease the weights in order to decrease the prediction error. ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐
36. 36. Updating Weights โข Each weight will be updated based on its derivative according to this equation: ๐พ๐๐๐๐ = ๐พ๐๐๐๐ โ ฮท โ ๐๐ฌ ๐๐พ๐ ๐พ ๐๐๐๐ = ๐พ ๐ โ ฮท โ ๐๐ฌ ๐๐พ ๐ = ๐. ๐ โ 0.01 โ ๐. ๐๐ ๐พ ๐๐๐๐ = ๐. ๐๐๐๐๐ ๐พ ๐๐๐๐ = ๐พ ๐ โ ฮท โ ๐๐ฌ ๐๐พ ๐ = ๐. ๐ โ 0.01 โ ๐. ๐๐๐ ๐พ ๐๐๐๐ = ๐. ๐๐๐๐ Updating ๐พ ๐ Updating ๐พ ๐ Continue updating weights according to derivatives and re-train the network until reaching an acceptable error.
37. 37. Second Example Backpropagation for NN with Hidden Layer
38. 38. ANN with Hidden Layer ๐พ ๐ ๐พ ๐ ๐พ ๐ ๐พ ๐ ๐พ ๐ ๐พ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐. ๐ ๐. ๐ ๐. ๐๐ ๐. ๐ โ๐. ๐ ๐. ๐ ๐. ๐ โ๐. ๐ ๐. ๐๐ ๐ ๐ ๐ ๐ ๐๐ฎ๐ญ๐ฉ๐ฎ๐ญ ๐. ๐ ๐. ๐ ๐. ๐๐ Training Data Initial Weights
39. 39. ANN with Hidden Layer Initial Weights PredictionTraining
40. 40. ANN with Hidden Layer Initial Weights PredictionTraining BackpropagationUpdate
41. 41. Forward Pass โ Hidden Layer Neurons ๐ ๐๐๐ = ๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐ = ๐. ๐ โ ๐. ๐ + ๐. ๐ โ ๐. ๐ + ๐. ๐ ๐ ๐๐๐ = ๐. ๐๐ ๐ ๐๐๐๐ = ๐ ๐ + ๐โ๐ ๐๐๐ = ๐ ๐ + ๐โ๐.๐๐ ๐ ๐๐๐๐ = ๐. ๐๐๐ ๐ ๐ In Out
42. 42. Forward Pass โ Hidden Layer Neurons ๐ ๐๐๐ = ๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐ = ๐. ๐ โ ๐. ๐๐ + ๐. ๐ โ ๐. ๐ โ ๐. ๐ ๐ ๐๐๐ = ๐. ๐๐๐ ๐ ๐๐๐๐ = ๐ ๐ + ๐โ๐ ๐๐๐ = ๐ ๐ + ๐โ๐.๐๐๐ ๐ ๐๐๐๐ = ๐. ๐๐๐ ๐ ๐ In Out
43. 43. Forward Pass โ Output Layer Neuron ๐๐๐๐๐ = ๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐ = ๐. ๐๐๐ โ โ๐. ๐ + ๐. ๐๐๐ โ ๐. ๐ + ๐. ๐๐ ๐๐๐๐๐ = ๐. ๐๐๐ ๐๐๐ ๐๐๐ = ๐ ๐ + ๐โ๐๐๐ ๐๐ = ๐ ๐ + ๐โ๐.๐๐๐ ๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐๐ In Out
44. 44. Forward Pass โ Prediction Error ๐๐๐๐๐๐๐ = ๐. ๐๐ ๐ฌ = ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐ ๐๐๐ ๐ = ๐ ๐ ๐. ๐๐ โ ๐. ๐๐๐ ๐ ๐ฌ = ๐. ๐๐๐ ๐ท๐๐๐๐๐๐๐๐ = ๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ , ๐๐ฌ ๐๐พ ๐ , ๐๐ฌ ๐๐พ ๐ , ๐๐ฌ ๐๐พ ๐ , ๐๐ฌ ๐๐พ ๐ , ๐๐ฌ ๐๐พ ๐
45. 45. Partial Derivatives Calculation
46. 46. Eโ๐5 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐พ ๐
47. 47. Eโ๐5 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐พ ๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐ ๐๐๐๐ ๐๐๐ ( ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐ ๐๐๐ ๐ ) = ๐ โ ๐ ๐ ๐๐๐๐๐๐๐ โ ๐๐๐ ๐๐๐ ๐โ๐ โ (๐ โ ๐) = ๐๐๐๐๐๐๐ โ ๐๐๐ ๐๐๐ โ (โ๐) ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐๐๐ ๐๐๐ โ ๐๐๐๐๐๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐๐๐ ๐๐๐ โ ๐๐๐๐๐๐๐ = ๐. ๐๐๐ โ ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐ Partial Derivative Substitution
48. 48. Eโ๐5 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐ ๐๐๐๐๐๐ ( ๐ ๐ + ๐โ๐๐๐ ๐๐ ) ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ( ๐ ๐ + ๐โ๐๐๐ ๐๐ )(๐ โ ๐ ๐ + ๐โ๐๐๐ ๐๐ ) ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ( ๐ ๐ + ๐โ๐.๐๐๐ )(๐ โ ๐ ๐ + ๐โ๐.๐๐๐ ) = ( ๐ ๐. ๐๐ )(๐ โ ๐ ๐. ๐๐ ) = ๐. ๐๐๐ ๐ โ ๐. ๐๐๐ = ๐. ๐๐๐ ๐. ๐๐๐ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ Partial Derivative Substitution
49. 49. Eโ๐5 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐พ ๐ (๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐) = ๐ โ ๐ ๐๐๐๐ โ (๐พ ๐) ๐โ๐ + ๐ + ๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐๐๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐๐๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐๐๐ Partial Derivative Substitution
50. 50. Eโ๐5 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐๐๐ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ โ ๐. ๐๐ โ ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐
51. 51. Eโ๐6 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐พ ๐
52. 52. Eโ๐6 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐
53. 53. Eโ๐6 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐พ ๐ (๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐) = ๐ + ๐ โ ๐ ๐๐๐๐ โ (๐พ ๐) ๐โ๐ +๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐๐๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐๐๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐๐๐ Partial Derivative Substitution
54. 54. Eโ๐6 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ โ ๐. ๐๐ โ ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐
55. 55. Eโ๐1 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
56. 56. Eโ๐1 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐
57. 57. Eโ๐1 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐ Partial Derivative Substitution ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐ ๐๐๐ ๐๐๐ (๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐) = (๐ ๐๐๐๐) ๐โ๐ โ ๐พ ๐ + ๐ + ๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐พ ๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐พ ๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = โ๐. ๐
58. 58. Eโ๐1 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐ Partial Derivative Substitution ๐๐๐ ๐๐๐ ๐๐๐๐๐ = ๐ ๐๐ ๐๐๐ ( ๐ ๐ + ๐โ๐ ๐๐๐ ) ๐๐๐ ๐๐๐ ๐๐๐๐๐ = ( ๐ ๐ + ๐โ๐ ๐๐๐ )(๐ โ ๐ ๐ + ๐โ๐ ๐๐๐ ) ๐๐๐ ๐๐๐ ๐๐๐๐๐ = ( ๐ ๐ + ๐โ๐ ๐๐๐ )(๐ โ ๐ ๐ + ๐โ๐ ๐๐๐ ) = ( ๐ ๐ + ๐โ๐.๐๐ )(๐ โ ๐ ๐ + ๐โ๐.๐๐ ) ๐๐ ๐๐๐๐ ๐๐ ๐๐๐ = ๐. ๐๐๐
59. 59. Eโ๐1 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐ Partial Derivative Substitution ๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐พ ๐ (๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐) = ๐ฟ ๐ โ (๐พ ๐) ๐โ๐+ ๐ + ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐ฟ ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐ฟ ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐
60. 60. Eโ๐1 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐ ๐๐ ๐๐๐๐ ๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = โ๐. ๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ โ ๐. ๐๐ โ โ๐. ๐ โ ๐. ๐๐๐ โ ๐. ๐ ๐๐ฌ ๐๐พ ๐ = โ๐. ๐๐๐
61. 61. Eโ๐2 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
62. 62. Eโ๐2 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐ ๐๐๐๐ ๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = โ๐. ๐
63. 63. Eโ๐2 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: Partial Derivative Substitution ๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐พ ๐ (๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐) = ๐ + ๐ฟ ๐ โ (๐พ ๐) ๐โ๐+๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐ฟ ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐ฟ ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
64. 64. Eโ๐2 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐ ๐๐๐๐ ๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = โ๐. ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ โ ๐. ๐๐ โ โ๐. ๐ โ ๐. ๐๐๐ โ ๐. ๐ ๐๐ฌ ๐๐พ ๐ = โ. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
65. 65. Eโ๐3 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
66. 66. Eโ๐3 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐
67. 67. Eโ๐3 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐ ๐๐๐ ๐๐๐ (๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐๐๐๐ โ ๐พ ๐ + ๐ ๐) = ๐ + (๐ ๐๐๐๐) ๐โ๐โ ๐พ ๐ + ๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐พ ๐ Partial Derivative Substitution ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐พ ๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐. ๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
68. 68. Eโ๐3 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐๐ ๐๐๐ ๐๐๐๐๐ = ๐ ๐๐ ๐๐๐ ( ๐ ๐ + ๐โ๐ ๐๐๐ ) ๐๐๐ ๐๐๐ ๐๐๐๐๐ = ( ๐ ๐ + ๐โ๐ ๐๐๐ )(๐ โ ๐ ๐ + ๐โ๐ ๐๐๐ ) Partial Derivative Substitution ๐๐๐ ๐๐๐ ๐๐๐๐๐ = ( ๐ ๐ + ๐โ๐ ๐๐๐ )(๐ โ ๐ ๐ + ๐โ๐ ๐๐๐ ) = ( ๐ ๐ + ๐โ๐.๐๐๐ )(๐ โ ๐ ๐ + ๐โ๐.๐๐๐ ) ๐๐ ๐๐๐๐ ๐๐ ๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
69. 69. Eโ๐3 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐พ ๐ (๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐) = ๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐ = (๐ฟ ๐) ๐โ๐โ ๐พ ๐ + ๐ + ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐พ ๐ Partial Derivative Substitution ๐๐๐๐๐ ๐๐พ ๐ = ๐พ ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
70. 70. Eโ๐3 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐. ๐ ๐๐ ๐๐๐๐ ๐๐ ๐๐๐ = ๐. ๐๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ โ ๐. ๐๐ โ ๐. ๐ โ ๐. ๐๐ โ ๐. ๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
71. 71. Eโ๐4 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
72. 72. Eโ๐4 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐. ๐ ๐๐ ๐๐๐๐ ๐๐ ๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
73. 73. Eโ๐4 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐๐๐๐ ๐๐พ ๐ = ๐ ๐๐พ ๐ (๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐) = ๐ฟ ๐ โ ๐พ ๐ + ๐ฟ ๐ โ ๐พ ๐ + ๐ ๐ = ๐ + (๐ฟ ๐) ๐โ๐โ ๐พ ๐ + ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐พ ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐พ ๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐ Partial Derivative Substitution ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
74. 74. Eโ๐4 ( ๐๐ฌ ๐๐พ ๐ ) Parial Derivative: ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ = ๐. ๐๐ ๐๐ฌ ๐๐๐๐ ๐๐๐ = ๐. ๐๐๐ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ = ๐. ๐ ๐๐ ๐๐๐๐ ๐๐ ๐๐๐ = ๐. ๐๐ ๐๐๐๐๐ ๐๐พ ๐ = ๐. ๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ โ ๐. ๐๐ โ ๐. ๐ โ ๐. ๐๐ โ ๐. ๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐๐ฌ ๐๐๐๐ ๐๐๐ โ ๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๐๐๐ โ ๐๐๐ ๐๐๐ ๐๐๐๐๐ โ ๐๐๐๐๐ ๐๐พ ๐
75. 75. All Error-Weights Partial Derivatives ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = โ. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = โ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐๐
76. 76. Updated Weights ๐พ ๐๐๐๐ = ๐พ ๐ โ ฮท โ ๐๐ฌ ๐๐พ ๐ = ๐. ๐ โ ๐. ๐๐ โ โ๐. ๐๐๐ = ๐. ๐๐๐๐๐ ๐พ ๐๐๐๐ = ๐พ ๐ โ ฮท โ ๐๐ฌ ๐๐พ ๐ = ๐. ๐ โ ๐. ๐๐ โ โ๐. ๐๐๐ = ๐. ๐๐๐๐๐ ๐พ ๐๐๐๐ = ๐พ ๐ โ ฮท โ ๐๐ฌ ๐๐พ ๐ = ๐. ๐๐ โ ๐. ๐๐ โ ๐. ๐๐๐ = ๐. ๐๐๐๐๐ ๐พ ๐๐๐๐ = ๐พ ๐ โ ฮท โ ๐๐ฌ ๐๐พ ๐ = ๐. ๐ โ ๐. ๐๐ โ ๐. ๐๐๐ = ๐. ๐๐๐๐ ๐พ ๐๐๐๐ = ๐พ ๐ โ ฮท โ ๐๐ฌ ๐๐พ ๐ = โ๐. ๐ โ ๐. ๐๐ โ ๐. ๐๐๐ = โ๐. ๐๐๐๐๐ ๐พ ๐๐๐๐ = ๐พ ๐ โ ฮท โ ๐๐ฌ ๐๐พ ๐ = ๐. ๐ โ ๐. ๐๐ โ ๐. ๐๐๐ = ๐. ๐๐๐๐๐ Continue updating weights according to derivatives and re-train the network until reaching an acceptable error.