Artificial neural networks (ANNs) or connectionist systems are a computational model used in machine learning, computer science and other research disciplines, which is based on a large collection of connected simple units called artificial neurons, loosely analogous to axons in a biological brain. Connections between neurons carry an activation signal of varying strength. If the combined incoming signals are strong enough, the neuron becomes activated and the signal travels to other neurons connected to it. Such systems can be trained from examples, rather than explicitly programmed, and excel in areas where the solution or feature detection is difficult to express in a traditional computer program. Like other machine learning methods, neural networks have been used to solve a wide variety of tasks, like computer vision and speech recognition, that are difficult to solve using ordinary rule-based programming. Find me on: AFCIT http://www.afcit.xyz YouTube https://www.youtube.com/channel/UCuewOYbBXH5gwhfOrQOZOdw Google Plus https://plus.google.com/u/0/+AhmedGadIT SlideShare https://www.slideshare.net/AhmedGadFCIT LinkedIn https://www.linkedin.com/in/ahmedfgad/ ResearchGate https://www.researchgate.net/profile/Ahmed_Gad13 Academia https://www.academia.edu/ Google Scholar https://scholar.google.com.eg/citations?user=r07tjocAAAAJ&hl=en Mendelay https://www.mendeley.com/profiles/ahmed-gad12/ ORCID https://orcid.org/0000-0003-1978-8574 StackOverFlow http://stackoverflow.com/users/5426539/ahmed-gad Twitter https://twitter.com/ahmedfgad Facebook https://www.facebook.com/ahmed.f.gadd Pinterest https://www.pinterest.com/ahmedfgad/

1. Artificial Neural Networks (ANNs)
XOR Step-By-Step
MENOUFIA UNIVERSITY
FACULTY OF COMPUTERS AND INFORMATION
ALL DEPARTMENTS
ARTIFICIAL INTELLIGENCE
‫المنوفية‬ ‫جامعة‬
‫والمعلومات‬ ‫الحاسبات‬ ‫كلية‬
‫األقسام‬ ‫جميع‬
‫الذكاء‬‫اإلصطناعي‬
‫المنوفية‬ ‫جامعة‬
Ahmed Fawzy Gad
ahmed.fawzy@ci.menofia.edu.eg

2. Classification Example
BA
01
1
10
00
0
11

3. Neural Networks
Input Hidden Output
BA
01
1
10
00
0
11

4. Neural Networks
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2

5. Neural Networks
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Can`t be Solved Linearly.
Single Layer Perceptron Can`t Work.
Use Hidden Layer.

6. Neural Networks
BA
01
1
10
00
0
11
Input OutputHidden

7. Input Layer
Input Output
A
B
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Hidden

8. Hidden Layer
Start by Two Neurons
Input Output
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Hidden
A
B

9. Output Layer
Input Output
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Hidden
1/0
𝒀𝒋
A
B

10. Weights
Input Output
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Hidden
Weights=𝑾𝒊
A
B
1/0
𝒀𝒋

11. Weights
Input Layer – Hidden Layer
Input Output
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Hidden
Weights=𝑾𝒊
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
A
B
1/0
𝒀𝒋

12. Weights
Hidden Layer – Output Layer
Input Output
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Hidden
Weights=𝑾𝒊
𝑾 𝟓
𝑾 𝟔
A
B
1/0
𝒀𝒋
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

13. All Layers
Input Output
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Hidden
Weights=𝑾𝒊
𝑾 𝟓
𝑾 𝟔
1/0
𝒀𝒋
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

14. Activation Function
Output
𝒀𝒋
BA
01
1
10
00
0
11
Input Hidden
1/0
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

15. Activation Function
Output
𝒀𝒋
BA
01
1
10
00
0
11
Input Hidden
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

16. Activation Function
Output
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

17. Activation Function
Components
Output
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

18. Activation Function
Inputs
Output
s
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

19. Activation Function
Inputs
Output
s
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
s=SOP(𝑿𝒊, 𝑾𝒊)

20. Activation Function
Inputs
Output
s
𝑿𝒊=Inputs 𝑾𝒊=Weights
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
s=SOP(𝑿𝒊, 𝑾𝒊)

21. Activation Function
Inputs
Output
s
𝑿𝒊=Inputs 𝑾𝒊=Weights
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
s=SOP(𝑿𝒊, 𝑾𝒊)

22. 𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
Activation Function
Inputs
Output
s
𝑿𝒊=Inputs 𝑾𝒊=Weights
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊
s=SOP(𝑿𝒊, 𝑾𝒊)
1/0

23. Activation Function
Inputs
Output
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊
1/0
Each Hidden/Output Layer
Neuron has its SOP.

24. Activation Function
Inputs
Output
s
𝑿 𝟏
𝑿 𝟐
𝑺 𝟏=(𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟑)
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊
1/0

25. Activation Function
Inputs
Output
s
𝑿 𝟏
𝑿 𝟐
𝑺 𝟐=(𝑿 𝟏 𝑾 𝟐+𝑿 𝟐 𝑾 𝟒)
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊
1/0

26. Activation Function
Inputs
Output
s
𝑿 𝟏
𝑿 𝟐
𝑺 𝟑=(𝑺 𝟏 𝑾 𝟓+𝑺 𝟐 𝑾 𝟔)
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊
1/0

27. Activation Function
Outputs
Output
F(s)s
𝑿 𝟏
𝑿 𝟐
Class Label
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

28. Activation Function
Outputs
Output
F(s)s
𝑿 𝟏
𝑿 𝟐
Class Label
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Input Hidden
1/0
Each Hidden/Output
Layer Neuron has its
Activation Function.
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

29. Activation Functions
Piecewise
Linear Sigmoid Binary

30. Activation Functions
Which activation function to use?
Outputs
Class
Labels
Activation
Function
TWO Class
Labels
TWO
Outputs
One that gives two outputs.
Which activation function to use?
𝑪𝒋𝒀𝒋
BA
01
1
10
00
0
11
BA
01
1 10
00
0 11

31. Activation Functions
Piecewise
Linear Sigmoid BinaryBinary

32. Activation Function
Output
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
BA
01
1
10
00
0
11
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
1/0
Input Hidden
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

33. Bias
Input Output
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

34. Bias
Hidden Layer Neurons
Input Output
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0

35. Bias
Output Layer Neurons
Input Output
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟑
1/0
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

36. All Bias Values
Input Output
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

37. Bias
Add Bias to SOP
Input Output
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
𝑺 𝟏=(𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟑)
𝑺 𝟐=(𝑿 𝟏 𝑾 𝟐+𝑿 𝟐 𝑾 𝟒)
𝑺 𝟑=(𝑺 𝟏 𝑾 𝟓+𝑺 𝟐 𝑾 𝟔)
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

38. Bias
Add Bias to SOP
Input Output
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
𝑺 𝟏=(𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟑)
1/0
+1
𝒃 𝟑
𝑺 𝟏=(+𝟏𝒃 𝟏+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟑)
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

39. Bias
Add Bias to SOP
Input Output
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
𝑺 𝟐=(𝑿 𝟏 𝑾 𝟐+𝑿 𝟐 𝑾 𝟒)
1/0
+1
𝒃 𝟑
𝑺 𝟐=(+𝟏𝒃 𝟐+𝑿 𝟏 𝑾 𝟐+𝑿 𝟐 𝑾 𝟒)
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

40. Bias
Add Bias to SOP
Input Output
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
𝑺 𝟑=(𝑺 𝟏 𝑾 𝟓+𝑺 𝟐 𝑾 𝟔)
1/0
+1
𝒃 𝟑
𝑺 𝟑=(+𝟏𝒃 𝟑+𝑺 𝟏 𝑾 𝟓+𝑺 𝟐 𝑾 𝟔)
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

41. Bias Importance
Input Output
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

42. Bias Importance
Input Output
X
Y
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

43. Bias Importance
Input Output
X
Y y=ax+b
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

44. Bias Importance
Input Output
X
Y y=ax+b
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

45. Bias Importance
Input Output
X
Y y=ax+b
Y-Intercept
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

46. Bias Importance
Input Output
X
Y y=ax+b
Y-Intercept
b=0
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

47. Bias Importance
Input Output
X
Y
Y-Intercept
b=0
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
y=ax+b

48. Bias Importance
Input Output
X
Y
Y-Intercept
b=+v
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
y=ax+b

49. Bias Importance
Input Output
X
Y
Y-Intercept
b=+v
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
y=ax+b

50. Bias Importance
Input Output
X
Y
Y-Intercept
b=-v
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
y=ax+b

51. Bias Importance
Input Output
X
Y
Y-Intercept
b=+v
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
y=ax+b

52. Bias Importance
Input Output
Same Concept Applies to Bias
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊+BIAS
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

53. Bias Importance
Input Output
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊+BIAS
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

54. Bias Importance
Input Output
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊+BIAS
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

55. Bias Importance
Input Output
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊+BIAS
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

56. Bias Importance
Input Output
S= 𝟏
𝒎
𝑿𝒊 𝑾𝒊+BIAS
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

57. Learning Rate
𝟎 ≤ η ≤ 𝟏
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

58. Summary of Parameters
Inputs 𝑿 𝒎
𝟎 ≤ η ≤ 𝟏
𝑿(𝒏)=(𝑿 𝟎, 𝑿 𝟏,𝑿 𝟐, …)
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

59. Summary of Parameters
Weights 𝑾 𝒎
𝟎 ≤ η ≤ 𝟏
W(𝒏)=(𝑾 𝟎, 𝑾 𝟏,𝑾 𝟐, …)
𝑿(𝒏)=(𝑿 𝟎, 𝑿 𝟏,𝑿 𝟐, …)
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

60. Summary of Parameters
Bias
𝟎 ≤ η ≤ 𝟏
𝑿(𝒏)=(𝑿 𝟎, 𝑿 𝟏,𝑿 𝟐, …)
W(𝒏)=(𝑾 𝟎, 𝑾 𝟏,𝑾 𝟐, …)
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

61. Summary of Parameters
Sum Of Products (SOP) 𝒔
s=(𝑿 𝟎 𝑾 𝟎+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟐+…)
𝟎 ≤ η ≤ 𝟏
𝑿(𝒏)=(𝑿 𝟎, 𝑿 𝟏,𝑿 𝟐, …)
W(𝒏)=(𝑾 𝟎, 𝑾 𝟏,𝑾 𝟐, …)
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

62. Summary of Parameters
Activation Function 𝒃𝒊𝒏
s=(𝑿 𝟎 𝑾 𝟎+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟐+…)
𝟎 ≤ η ≤ 𝟏
𝑿(𝒏)=(𝑿 𝟎, 𝑿 𝟏,𝑿 𝟐, …)
W(𝒏)=(𝑾 𝟎, 𝑾 𝟏,𝑾 𝟐, …)
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒

63. Summary of Parameters
Outputs 𝒀𝒋
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
s=(𝑿 𝟎 𝑾 𝟎+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟐+…)
𝟎 ≤ η ≤ 𝟏
𝑿(𝒏)=(𝑿 𝟎, 𝑿 𝟏,𝑿 𝟐, …)
W(𝒏)=(𝑾 𝟎, 𝑾 𝟏,𝑾 𝟐, …)

64. Summary of Parameters
Learning Rate η
𝟎 ≤ η ≤ 𝟏
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
s=(𝑿 𝟎 𝑾 𝟎+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟐+…)
𝑿(𝒏)=(𝑿 𝟎, 𝑿 𝟏,𝑿 𝟐, …)
W(𝒏)=(𝑾 𝟎, 𝑾 𝟏,𝑾 𝟐, …)

65. Other Parameters
Step n
𝒏 = 𝟎, 𝟏, 𝟐, …
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
s=(𝑿 𝟎 𝑾 𝟎+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟐+…)
𝟎 ≤ η ≤ 𝟏
𝑿(𝒏)=(𝑿 𝟎, 𝑿 𝟏,𝑿 𝟐, …)
W(𝒏)=(𝑾 𝟎, 𝑾 𝟏,𝑾 𝟐, …)

66. Other Parameters
Desired Output 𝒅𝒋
𝒏 = 𝟎, 𝟏, 𝟐, …
𝒅 𝒏 =
𝟏, 𝒙 𝒏 𝒃𝒆𝒍𝒐𝒏𝒈𝒔 𝒕𝒐 𝑪𝟏 (𝟏)
𝟎, 𝒙 𝒏 𝒃𝒆𝒍𝒐𝒏𝒈𝒔 𝒕𝒐 𝑪𝟐 (𝟎)
BA
01
1
10
00
0
11
F(s)s
𝑿 𝟏
𝑿 𝟐
bin
𝒀𝒋
+1
𝒃 𝟏
+1
𝒃 𝟐
1/0
+1
𝒃 𝟑
𝑾 𝟓
𝑾 𝟔
A
B
𝑾 𝟏
𝑾 𝟐
𝑾 𝟑
𝑾 𝟒
s=(𝑿 𝟎 𝑾 𝟎+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟐+…)
𝟎 ≤ η ≤ 𝟏
𝑿(𝒏)=(𝑿 𝟎, 𝑿 𝟏,𝑿 𝟐, …)
W(𝒏)=(𝑾 𝟎, 𝑾 𝟏,𝑾 𝟐, …)

67. Neural Networks Training Steps
Weights Initialization
Inputs Application
Sum of Inputs-Weights Products
Activation Function Response Calculation
Weights Adaptation
Back to Step 2
1
2
3
4
5
6

68. Regarding 5th Step: Weights Adaptation
• If the predicted output Y is not the same as the desired output d,
then weights are to be adapted according to the following equation:
𝑾 𝒏 + 𝟏 = 𝑾 𝒏 + η 𝒅 𝒏 − 𝒀 𝒏 𝑿(𝒏)
Where
𝑾 𝒏 = [𝒃 𝒏 , 𝑾 𝟏(𝒏), 𝑾 𝟐(𝒏), 𝑾 𝟑(𝒏), … , 𝑾 𝒎(𝒏)]

69. Neural Networks
Training Example
Step n=0
• In each step in the solution, the parameters of the neural network
must be known.
• Parameters of step n=0:
η = .001
𝑋 𝑛 = 𝑋 0 = +1, +1, +1,1, 0
𝑊 𝑛 = 𝑊 0 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6
= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1
𝑑 𝑛 = 𝑑 0 = 1
BA
01
1 => 1
10
00
0 => 0
11

70. Neural Networks
Training Example
Step n=0
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin
BA
01
1 => 1
10
00
0 => 0
11

71. Neural Networks
Training Example
Step n=0 – SOP – 𝑺 𝟏
𝑺 𝟏=(+𝟏𝒃 𝟏+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟑)
=+1*-1.5+1*1+0*1
=-.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

72. Neural Networks
Training Example
Step n=0 – Output – 𝑺 𝟏
𝒀 𝑺 𝟏 =
= 𝑩𝑰𝑵 𝑺 𝟏
= 𝑩𝑰𝑵 −. 𝟓
= 𝟎
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
𝟎, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

73. Neural Networks
Training Example
Step n=0 – SOP – 𝑺 𝟐
𝑺 𝟐=(+𝟏𝒃 𝟐+𝑿 𝟏 𝑾 𝟐+𝑿 𝟐 𝑾 𝟒)
=+1*-.5+1*1+0*1
=.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

74. Neural Networks
Training Example
Step n=0 – Output – 𝑺 𝟐
𝒀 𝑺2 =
= 𝑩𝑰𝑵 𝑺2
= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
−𝟏, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

75. Neural Networks
Training Example
Step n=0 – SOP – 𝑺 𝟑
𝑺 𝟑=(+𝟏𝒃 𝟑+𝑺 𝟏 𝑾 𝟓+𝑺 𝟐 𝑾 𝟔)
=+1*-.5+0*-2+1*1
=.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

76. Neural Networks
Training Example
Step n=0 – Output – 𝑺 𝟑
𝒀 𝑺3 =
= 𝑩𝑰𝑵 𝑺3
= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
𝟎, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

77. Neural Networks
Training Example
Step n=0 - Output
𝒀 𝒏 = 𝒀 𝟎 = 𝒀 𝑺3
= 1
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

78. Neural Networks
Training Example
Step n=0
Predicted Vs. Desired
𝒀 𝒏 = 𝒀 𝟎 = 1
𝐝 𝒏 = 𝒅 𝟎 = 1
∵ 𝒀 𝒏 = 𝒅 𝒏
∴ Weights are Correct.
No Adaptation
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

79. Neural Networks
Training Example
Step n=1
• In each step in the solution, the parameters of the neural network
must be known.
• Parameters of step n=1:
η = .001
𝑋 𝑛 = 𝑋 1 = +1, +1, +1,0, 1
𝑊 𝑛 = 𝑊 1 = 𝑊 0 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6
= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1
𝑑 𝑛 = 𝑑 1 = +1
BA
01
1 => 1
10
00
0 => 0
11

80. Neural Networks
Training Example
Step n=1
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin
BA
01
1 => 1
10
00
0 => 0
11

81. Neural Networks
Training Example
Step n=1 – SOP – 𝑺 𝟏
𝑺 𝟏=(+𝟏𝒃 𝟏+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟑)
=+1*-1.5+0*1+1*1
=-.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

82. Neural Networks
Training Example
Step n=1 – Output – 𝑺 𝟏
𝒀 𝑺 𝟏 =
= 𝑩𝑰𝑵 𝑺 𝟏
= 𝑩𝑰𝑵 −. 𝟓
= 𝟎
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
𝟎, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

83. Neural Networks
Training Example
Step n=1 – SOP – 𝑺 𝟐
𝑺 𝟐=(+𝟏𝒃 𝟐+𝑿 𝟏 𝑾 𝟐+𝑿 𝟐 𝑾 𝟒)
=+1*-.5+0*1+1*1
=.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

84. Neural Networks
Training Example
Step n=1 – Output – 𝑺 𝟐
𝒀 𝑺2 =
= 𝑩𝑰𝑵 𝑺2
= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
−𝟏, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

85. Neural Networks
Training Example
Step n=1 – SOP – 𝑺 𝟑
𝑺 𝟑=(+𝟏𝒃 𝟑+𝑺 𝟏 𝑾 𝟓+𝑺 𝟐 𝑾 𝟔)
=+1*-.5+0*-2+1*1
=.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

86. Neural Networks
Training Example
Step n=1 – Output – 𝑺 𝟑
𝒀 𝑺3 =
= 𝑩𝑰𝑵 𝑺3
= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
𝟎, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

87. Neural Networks
Training Example
Step n=1 - Output
𝒀 𝒏 = 𝒀 𝟏 = 𝒀 𝑺3
= 1
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

88. Neural Networks
Training Example
Step n=1
Predicted Vs. Desired
𝒀 𝒏 = 𝒀 𝟏 = 1
𝐝 𝒏 = 𝒅 𝟏 = 1
∵ 𝒀 𝒏 = 𝒅 𝒏
∴ Weights are Correct.
No Adaptation
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−2
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

89. Neural Networks
Training Example
Step n=2
• In each step in the solution, the parameters of the neural network
must be known.
• Parameters of step n=2:
η = .001
𝑋 𝑛 = 𝑋 2 = +1, +1, +1,0, 0
𝑊 𝑛 = 𝑊 2 = 𝑊 1 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6
= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1
𝑑 𝑛 = 𝑑 2 = 0
BA
01
1 => 1
10
00
0 => 0
11

90. Neural Networks
Training Example
Step n=2
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin
BA
01
1 => 1
10
00
0 => 0
11

91. Neural Networks
Training Example
Step n=2 – SOP – 𝑺 𝟏
𝑺 𝟏=(+𝟏𝒃 𝟏+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟑)
=+1*-1.5+0*1+0*1
=-1.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

92. Neural Networks
Training Example
Step n=2 – Output – 𝑺 𝟏
𝒀 𝑺 𝟏 =
= 𝑩𝑰𝑵 𝑺 𝟏
= 𝑩𝑰𝑵 −𝟏. 𝟓
= 𝟎
𝒃𝒊n 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
𝟎, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

93. Neural Networks
Training Example
Step n=2 – SOP – 𝑺 𝟐
𝑺 𝟐=(+𝟏𝒃 𝟐+𝑿 𝟏 𝑾 𝟐+𝑿 𝟐 𝑾 𝟒)
=+1*-.5+0*1+0*1
=-.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

94. Neural Networks
Training Example
Step n=2 – Output – 𝑺 𝟐
𝒀 𝑺2 =
= 𝑺𝑮𝑵 𝑺2
= 𝑺𝑮𝑵 −. 𝟓
=0
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
−𝟏, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

95. Neural Networks
Training Example
Step n=2 – SOP – 𝑺 𝟑
𝑺 𝟑=(+𝟏𝒃 𝟑+𝑺 𝟏 𝑾 𝟓+𝑺 𝟐 𝑾 𝟔)
=+1*-.5+0*-2+0*1
=-.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

96. Neural Networks
Training Example
Step n=2 – Output – 𝑺 𝟑
𝒀 𝑺3 =
= 𝑩𝑰𝑵 𝑺3
= 𝑩𝑰𝑵 −. 𝟓
= 𝟎
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
𝟎, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

97. Neural Networks
Training Example
Step n=2 - Output
𝒀 𝒏 = 𝒀 𝟐 = 𝒀 𝑺3
= 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

98. Neural Networks
Training Example
Step n=2
Predicted Vs. Desired
𝒀 𝒏 = 𝒀 𝟐 = 𝟎
𝐝 𝒏 = 𝒅 𝟐 = 𝟎
∵ 𝒀 𝒏 = 𝒅 𝒏
∴ Weights are Correct.
No Adaptation
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

99. Neural Networks
Training Example
Step n=3
• In each step in the solution, the parameters of the neural network
must be known.
• Parameters of step n=3:
η = .001
𝑋 𝑛 = 𝑋 3 = +1, +1, +1,1, 1
𝑊 𝑛 = 𝑊 3 = 𝑊 2 = 𝑏1, 𝑏2, 𝑏3, 𝑤1, 𝑤1, 𝑤2, 𝑤3, 𝑤4, 𝑤5, 𝑤6
= −1.5, −.5, −.5, 1, 1, 1, 1, −2, 1
𝑑 𝑛 = 𝑑 3 = 0
BA
01
1 => 1
10
00
0 => 0
11

100. Neural Networks
Training Example
Step n=3
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin
BA
01
1 => 1
10
00
0 => 0
11

101. Neural Networks
Training Example
Step n=3 – SOP – 𝑺 𝟏
𝑺 𝟏=(+𝟏𝒃 𝟏+𝑿 𝟏 𝑾 𝟏+𝑿 𝟐 𝑾 𝟑)
=+1*-1.5+1*1+1*1
=.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

102. Neural Networks
Training Example
Step n=3 – Output – 𝑺 𝟏
𝒀 𝑺 𝟏 =
= 𝑩𝑰𝑵 𝑺 𝟏
= 𝑩𝑰𝑵 . 𝟓
= 1
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
𝟎, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

103. Neural Networks
Training Example
Step n=3 – SOP – 𝑺 𝟐
𝑺 𝟐=(+𝟏𝒃 𝟐+𝑿 𝟏 𝑾 𝟐+𝑿 𝟐 𝑾 𝟒)
=+1*-.5+1*1+1*1
=1.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

104. Neural Networks
Training Example
Step n=3 – Output – 𝑺 𝟐
𝒀 𝑺2 =
= 𝑩𝑰𝑵 𝑺2
= 𝑩𝑰𝑵 𝟏. 𝟓
= 1
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
−𝟏, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

105. Neural Networks
Training Example
Step n=3 – SOP – 𝑺 𝟑
𝑺 𝟑=(+𝟏𝒃 𝟑+𝑺 𝟏 𝑾 𝟓+𝑺 𝟐 𝑾 𝟔)
=+1*-.5+1*-2+1*1
=-1.5
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

106. Neural Networks
Training Example
Step n=3 – Output – 𝑺 𝟑
𝒀 𝑺3 =
= 𝑩𝑰𝑵 𝑺3
= 𝑩𝑰𝑵 −𝟏. 𝟓
= 𝟎
𝒃𝒊𝒏 𝒔 =
+𝟏, 𝒔 ≥ 𝟎
𝟎, 𝒔 < 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

107. Neural Networks
Training Example
Step n=3 - Output
𝒀 𝒏 = 𝒀 𝟑 = 𝒀 𝑺3
= 𝟎
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

108. Neural Networks
Training Example
Step n=3
Predicted Vs. Desired
𝒀 𝒏 = 𝒀 𝟑 = 𝟎
𝐝 𝒏 = 𝒅 𝟑 = 𝟎
∵ 𝒀 𝒏 = 𝒅 𝒏
∴ Weights are Correct.
No Adaptation
BA
01
1 => 1
10
00
0 => 0
11
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin

109. Final Weights
s
𝑿 𝟏
𝑿 𝟐
𝒀𝒋
+1
−𝟏. 𝟓
+1
−. 𝟓
1/0
+1
−. 𝟓
−𝟐
+𝟏
A
B
+𝟏
+𝟏
+𝟏
+𝟏
bin
Current weights predicted
the desired outputs.