Slides for Moscow Data Science Meetup http://www.meetup.com/Moscow-Data-Science/events/230869938/

1. Deep learning and feature extraction for time
series forecasting
Pavel Filonov
pavel.filonov@kaspersky.com
27 May 2016

2. Outlines
Motivation
Cyber Physical Security
Problem formulation
Anomaly detection
Time series forecasting
Artificial Neural Networks
Basic model
RNN on raw data
Feature engineering
RNN on extracted features
Quasi-periodic timeseries
Conclusions

3. Cyber Physical Security
Image from http://www.wallpaperup.com

4. ”Pipeline” stand

5. Signal timeseries

6. Anomaly detection

7. Time series forecasting

8. Forecasting models
Auto-regression models and EMA (ARMA, ARIMA, GARCH)
Neural networks
Adaptive short term forecasting
Adaptive auto-regression
Adaptive model selection
Adaption model composition
Density forecast
Quantile regression
...

9. Neural networks for timeseries forecasting
Feed forward NN on window1
Recurrent NN
Hopfield networks
Elman networks
Long short term memory2
Gated Recurrent Unit3
1
https://www.cs.cmu.edu/afs/cs/academic/class/15782-
f06/slides/timeseries.pdf
2
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
3
http://arxiv.org/pdf/1406.1078v3.pdf

10. Neuron model
xi — inputs
b — bias
f — activation function
σ(t) = 1
1+e−t
tanh(t) = e2t
−1
e2t+1
f(t) = t
f(t) = H(t)
y — output
Figure: Single neuron

11. LSTM
ft = σ(Wf · [ht−1, xt] + bf )
it = σ(Wi · [ht−1, xt] + bi)
˜Ct = tanh(WC · [ht−1, xt] + bC)
Ct = ftCt−1 + it
˜Ct
ot = σ(Wo · [ht−1, xt] + bo)
ht = ot tanh(Ct)
Picture from: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

12. RNN on raw data
NN topology: 722 input → 64 LSTM + Dropout(0.2) → 722 Linear
Forecast horizon: 5 minutes

13. Timeseries segmentation
Segmentation
Features
extractionClustering
...
signal segments
Features matrix
Clusters Sequence of labels

14. RNN on extracted features
Let n be the number of clusters.
NN structure: n inputs → 10n LSTM → n SoftMax
Forecast horizon: 20 segments

15. Quasi-periodic timeseries

16. RNN on Quasi-periodic timeseries
NN structure:
61 → 32 LSTM+Dropout(0.2) → 64 LSTM+Dropout(0.2) → 1 Linear
Forecast horizon: 1 minute

17. Quasi-periodic timeseries
NN structure:
61 → 32 LSTM+Dropout(0.2) → 64 LSTM+Dropout(0.2) → 1 Linear
Forecast horizon: 1 minute

18. Conclusions
Picture from: http://www.simpsonscreative.co.uk/kiss-the-first-law-of-successful-copywriting/

19. References
http://keras.io/
https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2015-
56.pdf
Keras recurrent tutorial -
https://github.com/Vict0rSch/deep learning/tree/master/keras/recu
https://github.com/aurotripathy/lstm-anomaly-detect
https://github.com/aurotripathy/lstm-ecg-wave-anomaly-
detect
http://simaaron.github.io/Estimating-rainfall-from-weather-
radar-readings-using-recurrent-neural-networks/
http://danielhnyk.cz/predicting-sequences-vectors-keras-
using-rnn-lstm/