Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on Apache Spark
DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on Apache Spark
Similar to Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on Apache Spark (20)
More from DataWorks Summit/Hadoop Summit
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded
Recently uploaded (20)
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on Apache Spark
- 1. Unless stated otherwise all images are taken from wikipedia.org or openclipart.org Cognitive IoT Anomaly Detector with DeepLearning4J on IoT Sensor Data
- 3. Why IoT (now) ? • 15 Billion connected devices in 2015 • 40 Billion connected devices in 2020 • World population 7.4 Billion in 2016
- 17. Machine Learning on historic data Source: deeplearning4j.org
- 19. online vs. historic • Pros • low storage costs • real-time model update • Cons • algorithm support • software support • no algorithmic improvement • compute power to be inline with data rate • Pros • all algorithms • abundance of software • model re-scoring / re- parameterisation (algorithmic improvement) • batch processing • Cons • high storage costs • batch model update
- 22. Neural Networks
- 23. Neural Networks
- 25. Convolutional
- 27. Convolutional
- 29. Learning of a function A neural network can basically learn any mathematical function
- 30. Recurrent
- 31. LSTM “vanishing gradient problem” == influence of past inputs decay quickly over time
- 36. https://m.soundcloud.com/optometrist-prime/recurrence- music-written-by-a-recurrent-neural-network
- 37. •Outperformed traditional methods, such as •cumulative sum (CUSUM) •exponentially weighted moving average (EWMA) •Hidden Markov Models (HMM) •Learned what “Normal” is •Raised error if time series pattern haven't been seen before
- 40. Learning of an algorithm A LSTM network is Turing complete 1 1: http://binds.cs.umass.edu/papers/1995_Siegelmann_Science.pdf
- 41. Problems • Neural Networks are computationally very complex •especially during training •but also during scoring CPU (2009) GPU (2016) IBM TrueNorth (2017)
- 42. IBM TrueNorth •Scalable •Parallel •Distributed •Fault Tolerant •No Clock ! :) •IBM Cluster •4.096 chips •4 billion neurons •1 trillion synapses •Human Brain •100 billion neurons •100 trillion synapses •1.000.000 neurons •250.000.000 synapses
- 45. Watson IoT Watson Cognitive Services Analytics Model + API Driver Behaviour Personality Insights Trainable Model + API IoT for Insurance Visual Recognition Customizable Model + API Watson Machine Learning Data Science Platform as a Service Data Science Experience
- 46. DeepLearning4J
- 47. Components • DeepLearning4J Enterprise Grade DeepLearning Library • DataVec CSV/Audio/Video/Image/… => Vector • ND4J / ND4S (NumPy for the JVM)
- 48. ND4J • Tensor support (Linear Buffer + Stride) • Multiple implementations, one interface • vectorized c++ code (JavaCPP), off-heap data storage, BLAS (OpenBLAS, Intel MKL, cuBLAS) • GPU (CUDA 7.5)
- 49. turn on GPU
- 50. DL4J parallelisation • TensorFlow on ApacheSpark => • Scoring • Multi-model hyper-parameter tuning • Parallel training since V r0.8 • DeepLearning4J => • Scoring, Multi-model hyper-parameter tuning • Parallel training “Jeff Dean style parameter averaging”
- 51. “Code local vs spark” vs.
- 52. Demo IoT / Industry / Predictive Maintenance Use Case