Neural networks are composed of interconnected neurons arranged in layers that can learn patterns from data. They consist of an input layer, hidden layers, and an output layer. Each neuron receives weighted inputs, passes them through an activation function, and outputs the result. Backpropagation allows neural networks to learn by calculating error derivatives to update weights between layers. Deeper networks can model more complex patterns using techniques like convolutional neural networks for images and recurrent neural networks for sequential data. While powerful, neural networks require large datasets and computational resources to train effectively.

1. Intro to Deep Learning
Partners in Business IT Conference, USU, 3-24-16
by Adam Rogers, Data Scientist at Jane.com

2. What is a Neural Network?
http://promentumspeakers.com/what-happens-when-a-mental-illness-expert-has-an-
extremely-rare-stroke/
http://neuromastersoftware.com/neural-network-theory-introduction/

3. What is a Neural Network?
• Composed of Neurons
• Neurons have Activation Functions
• Neurons make up Layers
• Input Layer -> Hidden Layer(s) ->Output Layer

4. Neuron
Basic processing unit
Receives output of all neurons from
previous layer
Gets weighted sum and passes
through Activation Function to
calculate output
http://wwwold.ece.utep.edu/research/webfuzzy/docs/kk-thesis/kk-thesis-html/node14.html

5. Layer
• Where the magic happens
• Array of neurons that process outputs of previous layer (or input layer)
• Layers can have specific properties and activation functions

6. Activation Function
Performed on output of each neuron
Determines what (or whether) to pass on from each neuron to the next layer
Creates Non-linearity in your network
This is key to NN’s learning power!

7. Activation Functions
Sigmoid
http://www.saedsayad.com/artificial_neural_network.htm
Step
http://en.wikibooks.org/wiki/Artificial_Neural_Networks/Acti
vation_Functions#Continuous_Log-Sigmoid_Function
ReLU
http://cs231n.github.io/neural-networks-1/
And many others...

8. The Perceptron
The most basic Neural Network

9. Perceptron - Learning
• Gradient Descent (kind of)
• Run training example through network
• Calculate error of output and update weights
• Specifically:
• If output = 0 when it should be 1
• Add input vector to weight vector
• If output = 1 when it should be 0
• Subtract input vector from weight vector

10. Multilayer (Deep) Networks
• Perceptrons fail on complex problems (basically all real-world problems)
• Need better activation functions and more layers
• How to train (update weights of) multiple layers simultaneously?

11. Backpropagation!
• Most math-heavy part of NN’s
• Run training example through network and calculate error derivative on
output
• From there, calculate error derivatives of hidden unit outputs
• Then calculate error derivative on weights and update accordingly

12. Backpropagation
https://class.coursera.org/neuralnets-2012-001/lecture/39

13. Feel the Power
With hidden layers and better activation functions, NN’s work on far more
complicated problems
Non-linearity of activations allow untangling complicated data
http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

14. Convolutional NN’s
http://colah.github.io/posts/2014-
07-Conv-Nets-Modular/

15. Magical CNN’s
http://arxiv.org/pdf/1504.03293v1.pdf
Landrover!

16. CNN in Action!
http://demo.caffe.berkeleyvision.org/

17. Recurrent Neural Networks
• What if we have time series or sequential data where “memory” of prior data
points is relevant?
• Answering questions from an article
• What if we have variable input sizes?
• Text inputs
• What if we need variable output sizes?
• Captioning images
• Recurrent Neural Networks!

18. RNN’s
http://karpathy.github.io/assets/rnn/diags.jpeg

19. RNN Magic
http://arxiv.org/pdf/1411.4
389v3.pdf

20. RNN Magic
Ask Your Neurons: A Neural-based
Approach to Answering Questions
about Images
http://arxiv.org/pdf/1505.01121.pdf

21. RNN Magic
DRAW: A Recurrent Neural
Network For Image Generation
http://arxiv.org/pdf/1502.04623.pd
f

22. Deep Learning in NLP
How to represent relationships between words in a way that computers
understand?
Answer: Word Embeddings

23. Word Embeddings

24. Word Embeddings
http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/

25. Deep Learning in NLP
• Useful in many NLP techniques
• Sentiment Analysis
• Document Classification
• Question Analysis
• Language Translation
• Speech Recognition
• etc.

26. Tools
• Theano/Keras (Python tensor library)
• http://deeplearning.net/software/theano/
• http://keras.io/
• Caffe (Berkeley C++ deep learning library)
• http://caffe.berkeleyvision.org/
• Torch (Lua machine learning)
• https://github.com/dmlc/cxxnet
• TensorFlow from Google
• http://www.tensorflow.org/

27. Caffe
+ X 2

28. Keras

29. A Word of Warning
• While powerful, NN’s are very complicated
• Millions to billions of parameters can lead to overfitting
• Can take months to train (days on GPU)
• Not the answer to every problem
• Often a simpler algorithm will do almost as well or even better

30. Summary
• Neural Nets are a powerful and versatile tool
• Useful in many domains
• Broad and deep field of machine learning on its own
• Weigh complexity of NN’s against performance increase
• Many times simpler algorithms work almost as well

31. Thank you!