Deep Learning A-Z™: Artificial Neural Networks (ANN) - Module 1

1. © SuperDataScienceDeep Learning A-Z

2. © SuperDataScienceDeep Learning A-Z

3. © SuperDataScienceDeep Learning A-Z
What we will learn in this section:
• The Neuron
• The Activation Function
• How do Neural Networks work? (example)
• How do Neural Networks learn?
• Gradient Descent
• Stochastic Gradient Descent
• Backpropagation

4. © SuperDataScienceDeep Learning A-Z

5. © SuperDataScienceDeep Learning A-Z
Image Source: www.austincc.edu

6. © SuperDataScienceDeep Learning A-Z
Image Source: Wikipedia

7. © SuperDataScienceDeep Learning A-Z
Image Source: Wikipedia
Neuron
Axon
Dendrites

8. © SuperDataScienceDeep Learning A-Z
Image Source: Wikipedia

9. © SuperDataScienceDeep Learning A-Z
neuron
Node

10. © SuperDataScienceDeep Learning A-Z
Input signal 1
Input signal 2
Input signal m
neuron

11. © SuperDataScienceDeep Learning A-Z
Input signal 1
Input signal 2
Input signal m
Output signalneuron

12. © SuperDataScienceDeep Learning A-Z
neuron
Input value 1
Input value 2
Input value m
X1
X2
Xm
Output signal
Synapse

13. © SuperDataScienceDeep Learning A-Z
neuron
Input value 1
Input value 2
Input value m
X1
X2
Xm
y Output value
Synapse

14. © SuperDataScienceDeep Learning A-Z
neuron
X1
X2
Xm
Output value
Independent
variable 1
Independent
variable 2
Independent
variable m
Input value 1
Input value 2
Input value m
y
Standardize

15. © SuperDataScienceDeep Learning A-Z
Efficient BackProp
By Yann LeCun et al. (1998)
Link:
http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
Additional Reading:

16. © SuperDataScienceDeep Learning A-Z
neuron
Input value 1
Input value 2
Input value m
Output value
X1
X2
Xm
y Output value
Can be:
• Continuous (price)
• Binary (will exit yes/no)
• Categorical
Independent
variable 1
Independent
variable 2
Independent
variable m

17. © SuperDataScienceDeep Learning A-Z
neuron
Input value 1
Input value 2
Input value m
Output value 1
Output value 2
Output value p
X1
X2
Xm
yy2
y1
y3
Independent
variable 1
Independent
variable 2
Independent
variable m

18. © SuperDataScienceDeep Learning A-Z
neuron
Single Observation
Same observation
X1
X2
Xm
y
Input value 1
Input value 2
Input value m
Output value
Independent
variable 1
Independent
variable 2
Independent
variable m
Single Observation

19. © SuperDataScienceDeep Learning A-Z
neuron
Input value 1
Input value 2
Input value m
Output value
X1
X2
Xm
y
w1
w2
wm

20. © SuperDataScienceDeep Learning A-Z
?neuron
Input value 1
Input value 2
Input value m
y
X1
X2
Xm
w1
w2
wm
Output value

21. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
y
1st step:
X1
X2
Xm
w1
w2
wm
Output value

22. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
y
2nd step:
X1
X2
Xm
w1
w2
wm
Output value

23. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
y
2nd step:
X1
X2
Xm
w1
w2
wm
Output value
3rd step

24. © SuperDataScienceDeep Learning A-Z

25. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
y
2nd step:
X1
X2
Xm
w1
w2
wm
Output value
3rd step

26. © SuperDataScienceDeep Learning A-Z
y
0
1

27. © SuperDataScienceDeep Learning A-Z
0
y
Threshold Function
1

28. © SuperDataScienceDeep Learning A-Z
y
1
0
Sigmoid

29. © SuperDataScienceDeep Learning A-Z
0
y
Rectifier
1

30. © SuperDataScienceDeep Learning A-Z
0
y
1
-1
Hyperbolic Tangent (tanh)

31. © SuperDataScienceDeep Learning A-Z
Deep sparse rectifier
neural networks
By Xavier Glorot et al. (2011)
Link:
http://jmlr.org/proceedings/papers/v15/glorot11a/glorot11a.pdf
Additional Reading:

32. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
y
2nd step:
X1
X2
Xm
w1
w2
wm
Output value
3rd step
If threshold activation function:
If sigmoid activation function:
Assuming the DV is binary (y = 0 or 1)

33. © SuperDataScienceDeep Learning A-Z
y
4
5
6
7
Input value 1
Input value 2
Input value m
Output value
Input
Layer
Hidden
Layer
Output
Layer
X2
X1
Xm

34. © SuperDataScienceDeep Learning A-Z

35. © SuperDataScienceDeep Learning A-Z

36. © SuperDataScienceDeep Learning A-Z
.
Output Layer
Area (feet2)
Bedrooms
Distance to city (Miles)
Age
w1
w2
w3
w4
Price = w1*x1+ w2*x2+ w3*x3+ w4*x4
Input Layer
X4
X3
X2
X1
y

37. © SuperDataScienceDeep Learning A-Z
Input Layer
X4
X3
X2
X1Area (feet2)
Bedrooms
Distance to city (Miles)
Age
y
Hidden Layer Output Layer
.

38. © SuperDataScienceDeep Learning A-Z
Input Layer
X4
X3
X2
X1Area (feet2)
Bedrooms
Distance to city (Miles)
Age
y
Hidden Layer Output Layer
.

39. © SuperDataScienceDeep Learning A-Z
Input Layer
X4
X3
X2
X1Area (feet2)
Bedrooms
Distance to city (Miles)
Age
y
Hidden Layer Output Layer

40. © SuperDataScienceDeep Learning A-Z
Input Layer
Area (feet2)
Bedrooms
Distance to city (Miles)
Age
y
Hidden Layer Output Layer
X4
X3
X2
X1

41. © SuperDataScienceDeep Learning A-Z
Input Layer
Area (feet2)
Bedrooms
Distance to city (Miles)
Age
y
Hidden Layer Output Layer
X4
X3
X2
X1

42. © SuperDataScienceDeep Learning A-Z
Input Layer
Area (feet2)
Bedrooms
Distance to city (Miles)
Age
y
Hidden Layer Output Layer
X4
X3
X2
X1

43. © SuperDataScienceDeep Learning A-Z
Input Layer
Area (feet2)
Bedrooms
Distance to city (Miles)
Age
Hidden Layer Output Layer
X4
X3
X2
X1
y

44. © SuperDataScienceDeep Learning A-Z
Input Layer
Area (feet2)
Bedrooms
Distance to city (Miles)
Age
Hidden Layer Output Layer
X4
X3
X2
X1
y

45. © SuperDataScienceDeep Learning A-Z
Input Layer
Area (feet2)
Bedrooms
Distance to city (Miles)
Age
Hidden Layer Output Layer
X4
X3
X2
X1
y Price

46. © SuperDataScienceDeep Learning A-Z

47. © SuperDataScienceDeep Learning A-Z

48. © SuperDataScienceDeep Learning A-Z
y
Input value 1
Input value 2
Input value m
X1
X2
Xm
w1
w2
wm
Output value
y Actual value
ŷ

49. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
ŷ
X1
X2
Xm
w1
w2
wm
Output value
y Actual value
C = ½(ŷ- y)2
ŷ y C

50. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
Output value
Actual value
X1
X2
Xm
w1
w2
wm
ŷ
y
ŷ y C

51. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
X1
X2
Xm
w1
w2
wm
Output value
Actual value
ŷ
y
C = ½(ŷ- y)2
ŷ y C

52. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
X1
X2
Xm
w1
w2
wm
Output value
Actual value
ŷ
y
C = ½(ŷ- y)2
ŷ y C

53. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
X1
X2
Xm
w1
w2
wm
Output value
Actual value
ŷ
y
C = ½(ŷ- y)2
ŷ y C

54. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
X1
X2
Xm
w1
w2
wm
Output value
Actual value
ŷ
y
C = ½(ŷ- y)2
ŷ y C

55. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
X1
X2
Xm
w1
w2
wm
Output value
Actual value
ŷ
y
C = ½(ŷ- y)2
ŷ y C

56. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value m
X1
X2
Xm
w1
w2
wm
Output value
Actual value
ŷ
y
C = ½(ŷ- y)2
ŷ y C

57. © SuperDataScienceDeep Learning A-Z
C = ∑ ½(ŷ- y)2
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
ŷ y
C
ŷ y ŷ y ŷ y ŷ y ŷ y ŷ y ŷ y
Adjust w1, w2, w3

58. © SuperDataScienceDeep Learning A-Z
A list of cost functions used in
neural networks, alongside
applications
CrossValidated (2015)
Link:
http://stats.stackexchange.com/questions/154879/a-list-of-cost-functions-
used-in-neural-networks-alongside-applications
Additional Reading:

59. © SuperDataScienceDeep Learning A-Z

60. © SuperDataScienceDeep Learning A-Z
C = ∑ ½(ŷ- y)2
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
ŷ y
C
ŷ y ŷ y ŷ y ŷ y ŷ y ŷ y ŷ y
Adjust w1, w2, w3

61. © SuperDataScienceDeep Learning A-Z
yInput value w1 Output value
y Actual value
ŷ
C = ½(ŷ- y)2
X1

62. © SuperDataScienceDeep Learning A-Z
C = ½(ŷ- y)2
C
ŷ
Best!

63. © SuperDataScienceDeep Learning A-Z

64. © SuperDataScienceDeep Learning A-Z
Input Layer
Area (feet2)
Bedrooms
Distance to city (Miles)
Age
Hidden Layer Output Layer
X4
X3
X2
X1
y Price

65. © SuperDataScienceDeep Learning A-Z
Input Layer
Area (feet2)
Bedrooms
Distance to city (Miles)
Age
Hidden Layer Output Layer
y Price
X4
X3
X2
X1
25 weights

66. © SuperDataScienceDeep Learning A-Z
1,000 x 1,000 x … x 1,000 = 1,00025 = 1075 combinations
Sunway TaihuLight: World’s fastest Super Computer
93 PFLOPS
93 x 1015
1075 / (93 x 1015)
= 1.08 x 1058 seconds
= 3.42 x 1050 years

67. © SuperDataScienceDeep Learning A-Z
Image Source: neuralnetworksanddeeplearning.com

68. © SuperDataScienceDeep Learning A-Z

69. © SuperDataScienceDeep Learning A-Z
C = ½(ŷ- y)2
C
ŷ

70. © SuperDataScienceDeep Learning A-Z

71. © SuperDataScienceDeep Learning A-Z

72. © SuperDataScienceDeep Learning A-Z

73. © SuperDataScienceDeep Learning A-Z
C = ½(ŷ- y)2
C
ŷ

74. © SuperDataScienceDeep Learning A-Z
C
ŷ
Best!

75. © SuperDataScienceDeep Learning A-Z
C = ∑ ½(ŷ- y)2
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
ŷ y
C
ŷ y ŷ y ŷ y ŷ y ŷ y ŷ y ŷ y
Adjust w1, w2, w3

76. © SuperDataScienceDeep Learning A-Z
C = ∑ ½(ŷ- y)2
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
X1
X2
Xm
w1
w2
wm
ŷ
y
ŷ y
C
ŷ y ŷ y ŷ y ŷ y ŷ y ŷ y ŷ y
Adjust w1, w2, w3Adjust w1, w2, w3Adjust w1, w2, w3

77. © SuperDataScienceDeep Learning A-Z
Upd w’s
Upd w’s
Upd w’s
Upd w’s
Upd w’s
Upd w’s
Upd w’s
Upd w’s
Upd w’s

78. © SuperDataScienceDeep Learning A-Z
A Neural Network in 13 lines of
Python (Part 2 - Gradient
Descent)
Andrew Trask (2015)
Link:
https://iamtrask.github.io/2015/07/27/python-network-part2/
Additional Reading:

79. © SuperDataScienceDeep Learning A-Z
Neural Networks and Deep
Learning
Michael Nielsen (2015)
Link:
http://neuralnetworksanddeeplearning.com/chap2.html
Additional Reading:

80. © SuperDataScienceDeep Learning A-Z

81. © SuperDataScienceDeep Learning A-Z
Image Source: neuralnetworksanddeeplearning.com
Forward PropagationBackpropagation

82. © SuperDataScienceDeep Learning A-Z
Neural Networks and Deep
Learning
Michael Nielsen (2015)
Link:
http://neuralnetworksanddeeplearning.com/chap2.html
Additional Reading:

83. © SuperDataScienceDeep Learning A-Z
STEP 1: Randomly initialise the weights to small numbers close to 0 (but not 0).
STEP 2: Input the first observation of your dataset in the input layer, each feature in one input node.
STEP 3: Forward-Propagation: from left to right, the neurons are activated in a way that the impact of each neuron’s
activation is limited by the weights. Propagate the activations until getting the predicted result y.
STEP 4: Compare the predicted result to the actual result. Measure the generated error.
STEP 5: Back-Propagation: from right to left, the error is back-propagated. Update the weights according to how much
they are responsible for the error. The learning rate decides by how much we update the weights.
STEP 6: Repeat Steps 1 to 5 and update the weights after each observation (Reinforcement Learning). Or:
Repeat Steps 1 to 5 but update the weights only after a batch of observations (Batch Learning).
STEP 7: When the whole training set passed through the ANN, that makes an epoch. Redo more epochs.