This is an introductory workshop on Deep Learning with Keras. We start from shallow models: Linear Regression and Logistic Regression and show how they can be implemented with Keras. Then we show how to move to deeper models, how to use more complex architectures and layers. The workshop explores common use cases and suggests next steps to apply Keras to solve your problems.

1. Catalit LLC
INTRODUCTION
TO DEEP LEARNING
Francesco Mosconi
Data Weekends Catalit LLC

2. Catalit LLC
www.dataweekends.com/keras

3. Catalit LLC
MORNING
• Introduce Deep Learning
• Shallow Models
• Deep Models
• Convolutional Models

4. Catalit LLC
WHATTO EXPECT
• Accelerated learning
• Interactive
• Fun / Discovery
• Take notes

5. Catalit LLC
SUPERVISED LEARNING

6. Catalit LLC

7. Catalit LLC

8. Catalit LLC

9. Catalit LLC

10. Catalit LLC
LEARNING
Learning is not memorizing. It is generalizing the
conclusions to previously unseen examples

11. Catalit LLC
SUPERVISED LEARNING
Feature 1 Feature 2 Label
1 -0.14 0.18 Cat
2 1.07 -0.75 Dog
3 -2.58 0.63 Cat
… … … …

12. Catalit LLC
SUPERVISED LEARNING

13. Catalit LLC
PREDICTIONTYPE
http://i.ytimg.com/vi/WX0hnuniLpI/maxresdefault.jpg

14. Catalit LLC
PREDICTIONTYPE
CATEGORICAL CONTINUOUS
Eye colors Height of children
Courses at university Weight of cars
Highest degree Speed of the train
Gender Temperature
Spam or not Stock price

15. Catalit LLC
SUPERVISED LEARNING
CATEGORICAL CONTINUOUS
SUPERVISED CLASSIFICATION REGRESSION

16. Catalit LLC
LINEAR REGRESSION
Data Weekends Catalit LLC

17. Catalit LLC
Size in feet2 (x) Price in 1000$ (y)
2104 460
1416 232
1534 315
852 178
… …
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200
y = func(X)

18. Catalit LLC
Size in feet2 (x) Price in 1000$ (y)
2104 460
1416 232
1534 315
852 178
… …
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200
Hypothesis
ŷ = b+Xw

19. Catalit LLC
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200
Residual: yi - ŷi
ŷ = b+Xw

20. Catalit LLC
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200
Mean Squared Error
Residual: yi - ŷi
ŷ = b+Xw
COST FUNCTION

21. Catalit LLC
COST FUNCTION
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200
• Fixed data
• Fixed hypothesis
• Cost depends on values of
parameters

22. Catalit LLC
COST FUNCTION
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200
• Fixed data
• Fixed hypothesis
• Cost depends on values of
parameters

23. Catalit LLC
COST FUNCTION
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200
• Fixed data
• Fixed hypothesis
• Cost depends on values of
parameters

24. Catalit LLC
COST FUNCTION
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200
• Fixed data
• Fixed hypothesis
• Cost depends on values of
parameters

25. Catalit LLC
COST FUNCTION
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200
• Fixed data
• Fixed hypothesis
• Cost depends on values of
parameters

26. Catalit LLC
COST FUNCTION
Cost
0
1
2
3
4
Parameter
0 1 2 3 4
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200

27. Catalit LLC
OPTIMIZATION
Cost
0
1
2
3
4
Parameter
0 1 2 3 4
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200

28. Catalit LLC
OPTIMIZATION
Cost
0
1
2
3
4
Parameter
0 1 2 3 4
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200

29. Catalit LLC
OPTIMIZATION
Cost
0
1
2
3
4
Parameter
0 1 2 3 4
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200

30. Catalit LLC
OPTIMIZATION
Cost
0
4
8
12
16
Parameter
0 2 4 5 7
Pricein1000$(y)
100
200
300
400
500
Size in feet2 (x)
800 1150 1500 1850 2200

31. Catalit LLC
BEST MODEL
• Values of b and w that
give MINIMUM COST
Pricein
1000$
(y)
100
300
500
Size in feet2 (x)
800 1500 2200

32. Catalit LLC
MULTIVARIATE

33. Catalit LLC
CLASSIFICATION
Data Weekends Catalit LLC

34. Catalit LLC
BINARY CLASSIFICATION
Age Gender
Annual
Salary
Months in
residence
Months in
job
Current
Debt
Paid off
credit
Client 1 23 M $30,000 36 12 $5,000 Yes
Client 2 30 F $45,000 12 12 $1,000 Yes
Client 3 19 M $15,000 3 1 $10,000 No
Features Labels
Data Point

35. Catalit LLC
LOGISTIC REGRESSION

36. Catalit LLC
PERFORMANCE
Data Weekends Catalit LLC

37. Catalit LLC
BASELINE
• Know the score of the simplest model
• Compare your score with simplest model score

38. Catalit LLC
TRAIN /TEST
Model
Train
Model
Measure
performance
AllData
Typical values of test size: [10% - 50%]
Train
Test

39. Catalit LLC
REGRESSION SCORE
https://en.wikipedia.org/wiki/Coefficient_of_determination#/media/File:Coefficient_of_Determination.svg
Model

40. Catalit LLC
CONFUSION MATRIX
• Accuracy: Overall, how often is it correct?
• (TP +TN) / total
Test Negative Test Positive
Condition
Negative
TRUE NEGATIVE
FALSE POSITIVE
(Type I error)
Condition
Positive
FALSE NEGATIVE
(Type II error)
TRUE POSITIVE

41. Catalit LLC
CLASSIFICATION SCORES
• Accuracy: Overall, how often correct?
• (TP +TN) / total
• Precision: Test positive, how often prediction correct?
• TP / test yes
• Recall: Actual value positive, how often prediction correct?
• TP / actual yes
Test Negative Test Positive
Condition
Negative
TRUE NEGATIVE
FALSE POSITIVE
(Type I error)
Condition
Positive
FALSE NEGATIVE
(Type II error)
TRUE POSITIVE

42. Catalit LLC
LAB 01

43. Catalit LLC
DEEPER NETWORKS

44. Catalit LLC
Caption generation
Object recognition
http://www.image-net.org/challenges/LSVRC/

45. Catalit LLC
APPLICATIONS
Speech recognition and synthesis
http://venturebeat.com/2015

46. Catalit LLC
APPLICATIONS
Neural MachineTranslation
https://devblogs.nvidia.com/parallelforall/ https://smallbiztrends.com/2017/

47. Catalit LLC

48. Catalit LLC
https://deepmind.com www.nervanasys.com

49. Catalit LLC
https://www.pcper.com/category/tags/deep-neural-network

50. Catalit LLC
REASONS
Deep Learning
Traditional
Machine Learning
# training data
Performance

51. Catalit LLC
REASONS
Traditional
Machine Learning
# training data
Performance
Feature
extract
OutputFeatures
Shallow
ML algo
Input Deep Learning Output
Input
Deep Learning

52. Catalit LLC
FULLY CONNECTED
…
xN
x1
b
Input Output
b
b
…
b
b
Layer 1 Layer L
b
b
b
…
b
Layer 2
y
…

53. Catalit LLC
DEFINETHE NETWORK
• How many input units?
• How many output units?
• How many parameters?
• Which activation function?

54. Catalit LLC
REGRESSION
…
xN
x1
b
Input Output
b
b
…
b
Layer 1 Last layer
b
b
b
…
b
Layer 2
…
b
b
b
…
b
y
b
Relu Relu Relu

55. Catalit LLC
CLASSIFICATION
…
xN
x1
b
Input
Output
b
b
…
b
Layer 1 Last layer
b
b
b
…
b
Layer 2
…
b
b
b
…
b
y
b
Relu Relu Relu

56. Catalit LLC
MULTI CLASS
…
xN
x1
b
Input
Output
b
b
…
b
Layer 1 Last layer
b
b
b
…
b
Layer 2
…
b
b
b
…
b
A B C
1 0 0
1 0 0
0 1 0
0 0 1
Mutuall
y1
y2
y3
yJ
softmax
Relu Relu Relu

57. Catalit LLC
ACTIVATION FUNCTIONS

58. Catalit LLC
EPOCHS

59. Catalit LLC
MASTER FORMULA
model = data + structure + loss + optimizer

60. Catalit LLC
LAB 02

61. Catalit LLC
IMAGES

62. Catalit LLC
UNSTRUCTURED DATA
• Images
• Sound
• Text
• …
Feature 1 Feature 2 Feature 3
1 -0.7 1.34 -0.2
2 0.49 -1.44 -0.06
3 -1.36 -1.08 -1.15
… … … …
?

63. Catalit LLC
FEATURE ENGINEERING
http://www.privacysurgeon.org/
Distance 1 Distance 2 …
1 -0.7 1.34 …
2 0.49 -1.44 …
3 -1.36 -1.08 …
… … … …

64. Catalit LLC
FEATURE ENGINEERING
• FFT
• Wavelets
• …
Feature 1 Feature 2 …
1 -0.14 0.18 …
2 1.07 -0.75 …
3 -2.58 0.63 …
… … … …

65. Catalit LLC
FEATURE ENGINEERING
• Parts of speech
• Words frequency
• Embeddings

66. Catalit LLC
GRAYSCALE

67. Catalit LLC
MNIST
5 0 4 1

68. Catalit LLC
MNIST

69. Catalit LLC
MNIST

70. Catalit LLC
MNIST0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0.5 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0.5 0 1 1 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0.50.50.50.50.5 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0.5 1 1 0 0 0 0 0 1 1 0 0 1 1 0.5 0 0 0 0 0 0
0 0 0 0 0 0 0.5 1 1 0 0 0 0 0 1 1 0 0 0 0 0.5 1 1 1 0.5 0 0 0
0 0 0 0 0 0 1 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0.50.5 0 0 0 0
0 0 0 0 0 0 1 1 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0.5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
28 x 28 = 784

71. Catalit LLC
MNIST0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0.5 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0.5 0 1 1 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0.50.50.50.50.5 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0.5 1 1 0 0 0 0 0 1 1 0 0 1 1 0.5 0 0 0 0 0 0
0 0 0 0 0 0 0.5 1 1 0 0 0 0 0 1 1 0 0 0 0 0.5 1 1 1 0.5 0 0 0
0 0 0 0 0 0 1 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0.50.5 0 0 0 0
0 0 0 0 0 0 1 1 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0.5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

72. Catalit LLC
MNIST
0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 .5 .5 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 …
28 x 28 image => 784 input pixels array

73. Catalit LLC
FULLY CONNECTED
P
I
X
E
L
S
… …
…
…
0
1
9
Probability
of class
784 Inputs
10 outputs
with
SOFTMAX

74. Catalit LLC
CONVOLUTIONAL

75. Catalit LLC
LOCAL PATTERNS
• Fourier coefficients
• Wavelets
• Histogram of Oriented Gradients
(HOG)
• Speeded Up Robust Features (SURF)
• Local Binary Patterns (LBP)
• Color histograms
• …

76. Catalit LLC
TENSORS
Order Name Example
0 Scalar 3
1 Vector [4, 5, 0, 3, 1, 4, 5]
2 Matrix
[[0, 1, 0],
[5, 0, 2]]
3 Tensor
[[[0, 1, 0],
[5, 0, 2]],
[[1, 2, 4],
[8, 3, 1]]]

77. Catalit LLC
2 D CONV
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
=
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

78. Catalit LLC
1 -1 -1
-1 1 -1
-1 -1 1
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
=
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
-1 -1 1
-1 1 -1
1 -1 -1
1 -1 1
-1 1 -1
1 -1 1
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.11 0.33 -0.77 1.00 -0.77 0.33 -0.11
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
=
=
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1

79. Catalit LLC
CONVOLUTION LAYER
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.11 0.33 -0.77 1.00 -0.77 0.33 -0.11
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
1 -1 -1
-1 1 -1
-1 -1 1
1 -1 1
-1 1 -1
1 -1 1
-1 -1 1
-1 1 -1
1 -1 -1
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
Number of Filters Output Channels

80. Catalit LLC
INPUTTENSOR
Input: order 4 tensor
(N, H, W, C)
(60000, 28, 28, 1)
MNIST training set
N: Number of images
H: Height of image
W: Width of image
C: Number of color channels

81. Catalit LLC
1.00 0.33 0.55 0.33
0.33
MAX POOLING
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
maximum

82. Catalit LLC
1.00 0.33 0.55 0.33
0.33 1.00 0.33 0.55
0.55 0.33 1.00 0.11
0.33 0.55 0.11 0.77
MAX POOLING
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77

83. Catalit LLC
FEATURE EXTRACTION
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolution
ReLU
Pooling
Convolution
ReLU
Convolution
ReLU
Pooling
1.00 0.55
0.55 1.00
0.55 1.00
1.00 0.55
1.00 0.55
0.55 0.55
Layers can be repeated several (or many) times.

84. Catalit LLC
FLATTEN1.00 0.55
0.55 1.00
0.55 1.00
1.00 0.55
1.00 0.55
0.55 0.55
1.00
0.55
0.55
1.00
1.00
0.55
0.55
0.55
0.55
1.00
1.00
0.55

85. Catalit LLC
FULLY CONNECTED LAYER
X
O
1.00 0.55
0.55 1.00
0.55 1.00
1.00 0.55
1.00 0.55
0.55 0.55
1.00
0.55
0.55
1.00
1.00
0.55
0.55
0.55
0.55
1.00
1.00
0.55

86. Catalit LLC
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolution
ReLU
Pooling
Convolution
ReLU
Convolution
ReLU
Pooling
Fully
connected
Fully
connected
X
O
ALLTOGETHER

87. Catalit LLC
LAB 03

88. Catalit LLC
THANKYOU!
Francesco Mosconi
@framosconis info@catalit.com
www.dataweekends.com www.catalit.com
Data Weekends Catalit LLC