Deep Learning Class #1 - Go Deep or Go Home
•
6 likes•3,702 views
Slides by Louis Monier (Altavista Co-Founder & CTO) for Deep Learning keynote #1 at Holberton School. The keynote was followed by a workshop prepared by Gregory Renard. If you want to assist to similar keynote for free, checkout http://www.meetup.com/Holberton-School/
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Deep Learning Class #1 - Go Deep or Go Home
Similar to Deep Learning Class #1 - Go Deep or Go Home (20)
Recently uploaded
Recently uploaded (20)
Deep Learning Class #1 - Go Deep or Go Home
- 1. Louis Monier @louis_monier https://www.linkedin.com/in/louismonier Deep Learning Basics Go Deep or Go Home Gregory Renard @redo https://www.linkedin.com/in/gregoryrenard Class 1 - Q1 - 2016
- 2. Supervised Learning cathatdogcatmug … … =? prediction dog (89%) training set… Training (days) Testing (ms)
- 3. Single Neuron to Neural Networks
- 4. Single Neuron + x w1 xpos x w2 ypos w3 ChoiceScore Linear Combination Nonlinearity 3.4906 0.9981, red Inputs 3.45 1.21 -0.314 1.59 2.65
- 5. More Neurons, More Layers: Deep Learning input1 input2 input3 input layer hidden layers output layer circle iris eye face 89% likely to be a face
- 6. How to find the best values for the weights? w = Parameter Error a b Define error = |expected - computed| 2 Find parameters that minimize average error Gradient of error wrt w is Update: take a step downhill
- 7. Egg carton in 1 million dimensions Get to here or here
- 8. Backpropagation: Assigning Blame input1 input2 input3 label=1 loss=0.49 w1 w2 w3 b w1’ w2’ w3’ b’ 1). For loss to go down by 0.1, p must go up by 0.05. For p to go up by 0.05, w1 must go up by 0.09. For p to go up by 0.05, w2 must go down by 0.12. ... prediction p=0.3 p’=0.89 2). For loss to go down by 0.1, p’ must go down by 0.01 3). For p’ to go down by 0.01, w2’ must go up by 0.04
- 9. Stochastic Gradient Descent (SGD) and Minibatch It’s too expensive to compute the gradient on all inputs to take a step. We prefer a quick approximation. Use a small random sample of inputs (minibatch) to compute the gradient. Apply the gradient to all the weights. Welcome to SGD, much more efficient than regular GD!
- 10. Usually things would get intense at this point... Chain rule Partial derivatives Hessian Sum over paths Credit: Richard Socher Stanford class cs224d Jacobian matrix Local gradient
- 11. Learning a new representation Credit: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/
- 12. Training Data
- 13. Train. Validate. Test. Training set Validation set Cross-validation Testing set 2 3 41 2 3 41 2 3 41 2 3 41 shuffle! one epoch Never ever, ever, ever, ever, ever, ever use for training!
- 14. Tensors
- 15. Tensors are not scary Number: 0D Vector: 1D Matrix: 2D Tensor: 3D, 4D, … array of numbers This image is a 3264 x 2448 x 3 tensor [r=0.45 g=0.84 b=0.76]
- 16. Different types of networks / layers
- 17. Fully-Connected Networks Per layer, every input connects to every neuron. input1 input2 input3 input layer hidden layers output layer circle iris eye face 89% likely to be a face
- 18. Recurrent Neural Networks (RNN) Appropriate when inputs are sequences. We’ll cover in detail when we work on text. pipe a not is this t=0t=1t=2t=3t=4 this is not a pipe t=0 t=1 t=2 t=3 t=4
- 19. Convolutional Neural Networks (ConvNets) Appropriate for image tasks, but not limited to image tasks. We’ll cover in detail in the next session.
- 20. Hyperparameters Activations: a zoo of nonlinear functions Initializations: Distribution of initial weights. Not all zeros. Optimizers: driving the gradient descent Objectives: comparing a prediction to the truth Regularizers: forcing the function we learn to remain “simple” ...and many more
- 22. Sigmoid, tanh, hard tanh and softsign
- 23. ReLU (Rectified Linear Unit) and Leaky ReLU
- 24. Softplus and Exponential Linear Unit (ELU)
- 25. Softmax x1 = 6.78 x2 = 4.25 x3 = -0.16 x4 = -3.5 dog = 0.92536 cat = 0.07371 car = 0.00089 hat = 0.00003 Raw scores Probabilities S O F T M A X
- 26. Optimizers
- 27. Various algorithms for driving Gradient Descent Tricks to speed up SGD. Learn the learning rate. Credit: Alex Radford
- 29. Cross-entropy Loss x1 = 6.78 x2 = 4.25 x3 = -0.16 x4 = -3.5 dog = 0.92536 cat = 0.07371 car = 0.00089 hat = 0.00003 Raw scores Probabilities S O F T M A X dog: loss = 0.078 cat: loss = 2.607 car: loss = 7.024 hat: loss = 10.414 If label is: =? label (truth)
- 31. Overfitting Very simple. Might not predict well. Just right? Overfitting. Rote memorization. Will not generalize well.
- 32. Regularization: Avoiding Overfitting Idea: keep the functions “simple” by constraining the weights. Loss = Error(prediction, truth) + L(keep solution simple) L2 makes all parameters medium-size L1 kills many parameters. L1+L2 sometimes used.
- 33. At every step, during training, ignore the output of a fraction p of neurons. p=0.5 is a good default. Regularization: Dropout input1 input2 input3
- 34. One Last Bit of Wisdom
- 36. Workshop : Keras & Addition RNN https://github.com/holbertonschool/deep-learning/tree/master/Class%20%230