Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Deep learning frameworks v0.40 Slide 1 Deep learning frameworks v0.40 Slide 2 Deep learning frameworks v0.40 Slide 3 Deep learning frameworks v0.40 Slide 4 Deep learning frameworks v0.40 Slide 5 Deep learning frameworks v0.40 Slide 6 Deep learning frameworks v0.40 Slide 7 Deep learning frameworks v0.40 Slide 8 Deep learning frameworks v0.40 Slide 9 Deep learning frameworks v0.40 Slide 10 Deep learning frameworks v0.40 Slide 11 Deep learning frameworks v0.40 Slide 12 Deep learning frameworks v0.40 Slide 13 Deep learning frameworks v0.40 Slide 14 Deep learning frameworks v0.40 Slide 15 Deep learning frameworks v0.40 Slide 16 Deep learning frameworks v0.40 Slide 17 Deep learning frameworks v0.40 Slide 18 Deep learning frameworks v0.40 Slide 19 Deep learning frameworks v0.40 Slide 20 Deep learning frameworks v0.40 Slide 21 Deep learning frameworks v0.40 Slide 22 Deep learning frameworks v0.40 Slide 23 Deep learning frameworks v0.40 Slide 24 Deep learning frameworks v0.40 Slide 25 Deep learning frameworks v0.40 Slide 26 Deep learning frameworks v0.40 Slide 27 Deep learning frameworks v0.40 Slide 28 Deep learning frameworks v0.40 Slide 29 Deep learning frameworks v0.40 Slide 30 Deep learning frameworks v0.40 Slide 31 Deep learning frameworks v0.40 Slide 32 Deep learning frameworks v0.40 Slide 33 Deep learning frameworks v0.40 Slide 34
Upcoming SlideShare
Fundamental of deep learning
Next
Download to read offline and view in fullscreen.

2 Likes

Share

Download to read offline

Deep learning frameworks v0.40

Download to read offline

ODSC March Meetup

Related Books

Free with a 30 day trial from Scribd

See all

Deep learning frameworks v0.40

  1. 1. Deep Learning - Concepts & Frameworks Peter Morgan – Data Science Partnership Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 1
  2. 2. Contents • Deep Learning Concepts ………….3 • Deep Learning Frameworks …..26 • Specific Frameworks ………..……30 • Comparison …………………………..33 • Questions ………………………………34 Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 2
  3. 3. Overview - Concepts not Code No Maths No Code Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 3
  4. 4. Frameworks - The Big Picture Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 4 AI Frameworks Cognitive Architectures ML Frameworks Supervised, Unsupervised & Reinforcement Deep Learning Frameworks Neural Nets
  5. 5. What is Deep learning? • Deep learning refers to algorithms based on artificial neural networks (ANNs), which in turn are based on biological neural networks (BNN), such as the human brain. • In practice it consists of mulitple layers, nodes, weights and optimisation algorithms • Due to more labeled data, more compute power, better optimization algorithms, and better neural net models and architectures, deep learning has started to supersede humans when it comes to image recognition and classification. • Work is being done to obtain similar levels of performance in natural language processing and understanding. • Deep learning applies to supervised, unsupervised and reinforcement learning. • According to Jeff Dean in a recent interview, Google have implemented DL in over one hundred of their products and services including search and photos. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 5
  6. 6. Deep Learning Evolution Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 6
  7. 7. Deep Learning Concepts Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 7
  8. 8. Data Sets • Raw data input into the neural network can originate from any environmental source. • It can be recorded and stored in a database (e.g., text, images, audio, video) or live (incident directly from the environment), so called streaming data. • Examples of recorded data sets include the MNIST and Labeled Faces in the Wild (LFW). Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 8 MNIST LFW
  9. 9. Multilayer Perceptron (MLP) • A MLP is a feedforward artificial neural network model that maps sets of input data onto a set of appropriate outputs (Rosenblatt, 1958). • An MLP consists of multiple layers of nodes with each layer fully connected to the next one. • Except for the input nodes, each node is a neuron (or processing element) with a nonlinear activation function. • MLP utilizes a supervised learning technique called backpropagation for training the network. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 9
  10. 10. Layers and Nodes • Layers • A neural network is made up of layers of nodes. • There is an input (visible) layer, an output (classification) layer, and several hidden layers. • A typical NN may have between 10 and 30 layers, but sometimes many more. • Every layer of a deep learning network requires four elements: the input (vector), the weights (matrix), a bias and the transform (activation function). • Nodes • Nodes in a neural network represent the places where calculations are done. • At each node, the input values xi are multiplied by a weight wi, summed, added to a bias b, and fed into an activation function. • A decision is then made whether to transmit the resultant value depending on if it exceeds a certain threshold value or not. • For example, images from the MNIST data set have 784 pixels, so neural nets processing them must have 784 input nodes, one per pixel. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 10
  11. 11. Weights & Softmax • Weights • Coefficients that amplify or mute the input signal coming into each node. • They are assigned initial values which can be random or chosen based upon some insight. • A NN can be represented by its weight matrix along with its activation functions. • Softmax • The softmax function, or normalized exponential, is a generalization of the sigmoid logistic function. • In neural network simulations, the softmax function is often implemented at the final layer of a network used for classification. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 11
  12. 12. Activation Function Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 12 • One of a set of functions that determine the threshold at each node above which a signal is passed through the node, and below which it is blocked. • Activation functions normalize input from a previous layer to a value between -1 and 1 to ensure an output layer probability of between 0 and 1. • Functions that achieve this include the logistic, sigmoid, tanh and ReLU functions.
  13. 13. Optimization and Overfitting • Optimization • Refers to the manner by which a neural net minimizes error as it adjusts its coefficients (weights) step by step. • L-BFGS is one such algorithm. • Overfitting • Overfitting is where too many parameters are used in the model constructed to fit the data which leads to poor predictive power of the model. • Regularisation, cross-validation and dropout are all methods used to address this problem. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 13
  14. 14. Regularisation and Cross-validation • Both techniques are used to prevent overfitting • Regularisation • Regularisation refers to a process of introducing additional information in order to prevent overfitting. • It penalizes models with extreme parameter values by introducing a factor which weights the penalty against more complex models with an increasing variance in the data errors. • Cross-validation • Cross-validation combines sampling sets to correct for overfitting and derive a more accurate estimate of model prediction performance. • One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). • To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 14
  15. 15. Feed Forward Networks • The feedforward neural network was the first and simplest type of artificial neural network devised. • In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes and to the output nodes. • There are no cycles or feedback loops in the network. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 15
  16. 16. Support Vector Machines (SVM) • Support vector machines are supervised learning algorithms that analyze data and recognize patterns, in order to classify data points (Vapnik, 1979). • Given a set of training examples, each marked for belonging to one of two categories, an SVM training algorithm builds a model that assigns new examples into one category or the other – it is a linear classifier. • Recently SVM’s have been usurped by the success of deep learning algorithms. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 16
  17. 17. Stochastic Gradient Descent • Stochastic gradient descent is a popular algorithm for training a wide range of models in machine learning, including support vector machines and logistic regression. • When combined with the backpropagation algorithm, it is the de facto standard algorithm for training artificial neural networks. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 17
  18. 18. Back Propagation • Backpropagation is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent. • The method calculates the gradient of a loss function with respect to all the weights in the network. • The gradient is fed to the optimization method which in turn uses it to update the weights, in an attempt to minimize the loss function. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 18
  19. 19. Markov Models • Markov Model • A Markov model is a stochastic (probabilistic) model used to model randomly changing systems where it is assumed that future states depend only on the present state and not on the sequence of events that preceded it. • Markov Chain • Markov Chains are essentially logical circuits that connect two or more states via probabilities. • Markov Chains are sequential. Their purpose is to give you a good idea, given one state, of what the next state will be. • Use cases include natural language processing (NLP) and stock price trading. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 19
  20. 20. Hidden Markov Model • An HMM is a statistical Markov model in which the system being modeled is assumed to be a Markov process with unobserved (hidden) states. • A HMM can be presented as the simplest dynamic Bayesian network. • They are used in time series prediction, e.g., in language analysis. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 20
  21. 21. Restricted Boltzmann Machines & Autoencoders • Restricted Boltzmann Machines (RBM) • Invented by Geoff Hinton (1999) at the University of Toronto, RBMs are shallow, two- layer neural nets consisting of an input layer and a hidden layer. • They constitute the building blocks of deep neural networks - a deep learning network is simply many restricted Boltzmann machines stacked on top of one another. • Nodes are connected to each other across layers, but have the restriction that no two nodes of the same layer are linked. • Autoencoder • An autoencoder is an ANN used for learning efficient codings. • The aim of an autoencoder is to learn a compressed, distributed representation (encoding) for a set of data, typically for the purpose of dimensionality reduction. • In an autoencoder, the output layer has equally many nodes as the input layer, and instead of training it to predict some target value y given inputs x, an autoencoder is trained to reconstruct its own inputs x. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 21
  22. 22. Max Pooling and Cost Functions • Max Pooling • In CNN’s, max-pooling partitions the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum value. • Loss Function • A loss function or cost function is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cost" associated with the event. • The opposite of a loss function is called a reward function, or utility function. • Dropout The dropout method is introduced to prevent overfitting. • At each training stage, individual nodes are either "dropped out" of the net with probability 1-p or kept with probability p, so that a reduced network is left. • By avoiding training all nodes on all the training data, dropout decreases overfitting in neural nets. • The method also significantly improves the speed of training. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 22
  23. 23. Convolutional Neural Networks • First developed in 1970’s. • Widely used for image recognition and classification. • Inspired by biological processes, CNN’s are a type of feed-forward ANN. • The individual neurons are tiled in such a way that they respond to overlapping regions in the visual field. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 23
  24. 24. Recurrent Neural Networks • First developed in 1970’s. • RNN’s are neural networks that are used to predict the next element in a sequence or time series. • This could be, for example, words in a sentence or letters in a word. • Applications include predicting or generating music, stories, news, code, financial instrument pricing, text, speech, in fact the next element in any event stream. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 24
  25. 25. LSTM and NTM • Long Short Term Memory (LSTM) • LSTM is an RNN architecture that contains blocks that can remember a value for an arbitrary length of time. • It solves the vanishing or exploding gradient problem when calculating back propagation. • An LSTM network is universal in the sense that given enough network units it can compute anything a conventional computer can compute, provided it has the proper weight matrix. • LSTM outperforms alternative RNNs and Hidden Markov Models and other sequence learning methods in numerous applications, e.g., in handwriting recognition, speech recognition and music composition. • Neural Turing Machines (NTM) • NTMs are a method of extending the capabilities of recurrent neural networks by coupling them to external memory resources. Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 25
  26. 26. Deep Learning Frameworks Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 26 Framework = Toolkit = Library
  27. 27. Deep Learning Frameworks • Apache SINGA • Brainstorm • Caffe • Chainer • CNTK (Microsoft) • DL4J • DMLC • Fbcunn (Facebook) • Lasagne • Minerva • Mocha.jl (Julia) • MXnet • Neon (Nervana) • Purine • Tensorflow (Google) • Theano • Torch • Warp-CTC (Baidu) Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 27
  28. 28. Deep Learning Frameworks (cont) • Brain (Javascript) • Cudamat • Deep Learning Framework (Intel) • Deepnet • Hebel • Infer.NET • Keras • Leaf • MLPNeuralNet • Neural Network Toolbox (MatLab) • Neuraltalk • Neurolab • OpenDeep • PyBrain • Swift-AI • VELES (Samsung) Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 28
  29. 29. Some Specific DL Libraries • ANN • Brainstorm • clab/cnn • DeepHear (music composition) • DoctorTeeth/diffmem • miloharper/neural-network-animation • CNN • ConvNetJS • Marvin • MatConvNet • RNN • awesome-rnn • karpathy/char-rnn • karpathy/neuraltalk2 • wojciechz/learning_to_execute • LSTM • dl4j-0.4 Graves LTSM • github.nicodjimenez/lstm • Russell91/LSTMSummation Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 29
  30. 30. TensorFlow • TensorFlow is the newly (Nov 2015) open sourced deep learning library from Google. • It is their second generation system for the implementation and deployment of large-scale machine learning models. • Written in C++ with a python interface, it is borne from research and deploying machine learning projects throughout a wide range of Google products and services. • Initially TF ran only on a single node (your laptop, say), but Google have recently released a version that now runs on a cluster. • https://www.tensorflow.org/ Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 30
  31. 31. Torch • First released in 2000, with over 50,000 downloads, company users include Google, Facebook, Twitter. • The goal of Torch is to have maximum flexibility and speed in building scientific algorithms while making the process extremely simple. • Torch is a neural network library written in Lua with a C/CUDA interface originally developed by a team from the Swiss institute EPFL. • At the heart of Torch are popular neural network and optimization libraries which are simple to use, while being flexible in implementing different complex neural network topologies. • http://torch.ch/ Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 31
  32. 32. Theano • Theano is a deep learning library written in python and popular for its ease of use. • Using Theano, it is possible to attain speeds rivalling hand-crafted C implementations for problems involving large amounts of data. • Theano has been powering large-scale computationally intensive scientific investigations since 2007 • Supports CuDNN v3 • http://deeplearning.net/software/theano/ Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 32
  33. 33. DL Framework Comparison • The most common metrics we can use to compare these deep learning frameworks are: • Speed of execution • Ease of use • Languages used (core and front end) • Resources (CPU and memory capacity) needed in order to run the various algorithms • GPU support • Size of active community of users • Contributors and committers • Platforms supported (e.g., OS, single devices and/or distributed systems) • Algorithmic support • Number of packages in their library • For example, various benchmarks and comparisons are available here: https://github.com/soumith/convnet-benchmarks Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 33
  34. 34. Questions? Deep Learning Frameworks ODSC Meetup Peter Morgan 16 Mar 2016 34 www.datasciencepartnership.com @insight_ai
  • ananp11

    May. 20, 2016
  • OlivierVin

    Apr. 6, 2016

ODSC March Meetup

Views

Total views

1,379

On Slideshare

0

From embeds

0

Number of embeds

9

Actions

Downloads

92

Shares

0

Comments

0

Likes

2

×