Time series is prevalent in the IoT environment and used for monitoring the evolving behavior of involved entities or objects over time. Analyzing and mining such time series data serve for revealing insightful long-term and instantaneous information behind the data, e.g., trend, event, correlation and causality and so on. Inspired by the recent successes of neural networks, in this talk we present a novel end-to-end hybrid neural network for learning the local and global contextual features of time series. In particular, we explore the idea of hybrid neural networks in a specific time series learning problem, namely learning the local trend of time series. Local trends of time series characterize the intermediate upward and downward patterns of time series. Learning and forecasting the local trend in time series data play an important role in many real applications, ranging from investing in the stock market, resource allocation in data centers and load schedule in the smart grid. We propose TreNet, a hybrid neural network which leverages convolutional neural networks (CNNs) to extract salient features from local raw data of time series and a long-short term memory recurrent neural network (LSTM) to capture such dependency of local trends. Preliminary experimental results on real datasets demonstrate the superiority of TreNet over conventional CNN, LSTM, HMM method and various kernel based baselines.

1. Hybrid Neural Networks for
Time Series Learning
Tian Guo
Ph.D.
24 Nov. 2016

2. Outline
• Introduction
• Hybrid Neural Network (HNN)
• HNN for Real
• Preliminaries
• TreNet: a HNN for learning the local trend of time series
• Experiment results
• Conclusion
2

3. Introduction
3Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

4. Introduction
• Time series data: a sequence of data points consisting of
successive measurements made over time.
• Internet of Things
• Sensor networks
• Mobile phones
• And more…
4Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

5. Introduction
• Time series analytics in a variety of applications
• Classification
• Prediction
• Anomaly detection
• Pattern discovery
• And more…
5
Pattern 1 Pattern 2
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

6. Introduction
• Time series analytics tools
• Statistics
• Hidden Markov Model (HMM)
• State Space Model
• ARIMA
• Machine learning
• Random Forest
• SVM
• Gaussian Process
•
6
Random Forest
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

7. Introduction
• Neural Network and Deep Learning
• Language translation
• E.g., Google’s Multilingual Neural Machine Translation System
• Computer vision
• E.g., Microsoft Research’s PReLU network outperforms Human-Level
performance on ImageNet Classification
• Speech recognition
• E.g., Amazon Echo, Apple Siri
• Time series classification
• E.g. recognize patterns in multivariate
time series of clinical measurements
7
Z. Lipton, et al. “Learning to Diagnose with LSTM Recurrent Neural Networks”. ICLR, 2016.
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

8. Introduction
• Fundamental network architectures
• Convolutional neural network: input two-dimensional data, e.g.,
image
• Recurrent neural network: input
8
Unfolded recurrent connections in a RNN
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

9. Introduction
• Convolutional Neural Network (CNN)
• Feature learning for images
• To extract high-level features from raw data
• Such high-level features are further used for classification or
regression.
9
K. He, et al. “Delving deep into rectifiers: Surpassing Human-Level Performance on ImageNet Classification”. arxiv.org, 2015
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

10. Introduction
• Convolutional Neural Network (CNN)
• CNN in the Human Activity Recognition Problem
• Multichannel time series acquired from a set of body-worn sensors
• To predict human activities by training a CNN over time series
10
J. Yang, et al. “Deep Convolutional Neural Networks On Multichannel Time Series For Human Activity Recognition”. IJCAI 2015
Value
Time
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

11. Introduction
• Recurrent Neural Network (RNN)
• A powerful tool to model sequence data
• To capture dependency in sequence data
• Long-short term memory network (LSTM)
• A widely used variant of RNNs
• Equipped with memory and gate mechanism
• To overcome gradient vanishing and explosion
11
S. Hochreiter, et al. “Long short-term memory’’. Neural computation, 1997.
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

12. Hybrid Neural Networks
12Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

13. Hybrid Neural Networks
• CNN or RNN
• Work well for respective data, i.e. images and sequence data
13Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

14. Hybrid Neural Networks
• A cascade of CNN and RNN
• Classification of EEG data
• Electroencephalogram (EEG) : multiple time series corresponding to
measurements across different spatial locations over the cortex.
• Mental load classification task:
measures the working memory
responsible for transient retention
of information in the brain.
14
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

15. Hybrid Neural Networks
• A cascade of CNN and RNN
• Classification of EEG data
• A key challenge in correctly recognizing mental states
• EEG data often contains translation
and deformation of signal in space,
frequency, and time, due to inter-
and intra-subject differences
15
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

16. Hybrid Neural Networks
• A cascade of CNN and RNN
• Classification of EEG data
16
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

17. Hybrid Neural Networks
• A cascade of CNN and RNN
• EEG data classification
17
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

18. Hybrid Neural Networks
• Time Series
• Noisy
• Non-stationary
• Hidden information: states, dynamics
• Auto-correlated on the temporal dimension
• Manual feature engineering
• Preprocessing: de-trending, outlier removal, etc.
• Dimension reduction: Fourier Transformation
• Piecewise approximation: PAA, PCA, PLA, etc.
• Application-specific, domain knowledge
18
Why do we need hybrid architectures?
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

19. Hybrid Neural Networks
• Hybrid architectures: end-to-end learning framework
• Loss function driven training
• Learning representative features
• Capturing sequential dependency in data
19
Why do we need hybrid architectures?
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

20. Hybrid Neural Networks
• A cascade of CNN and RNN
20
P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

21. HNN for Real:
TreNet for Learning the Local Trend
21
T. Guo, et al. TreNet: Hybrid Neural Networks for Learning the Local Trend in Temporal Data. In submission to ICLR, 2017
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

22. Preliminaries
• Conventional trend analysis in time series
is the seasonal component at time t
is the trend component at t
is the remainder
22Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

23. Preliminaries
• Local trend
• Measure the intermediate local behaviour, i.e. upward or downward
pattern of time series
• For instance, the time series of household power consumption and
the local trends are shown as follows:
• Time series
• Extracted local trend ,
is the duration and is the slope
23Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

24. Preliminaries
• Learning and forecasting the local trend
• Predict ,
• Useful in many applications
• Smart energy
• Stock market
• And more …
24Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

25. Preliminaries
• Learning and forecasting the local trend
• Local raw data
• Global contextual information
in the historical sequence of trend
• To learn a function
is either or
25Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

26. TreNet
• Overview of TreNet
• is derived by training the LSTM over sequence to
capture the dependency in the trend evolving.
• corresponds to local features extracted by CNN from
26Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

27. is derived by training the LSTM over sequence to capture
the dependency in the trend evolving.
corresponds to local features extracted by CNN from
TreNet
• Overview
•
27Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

28. TreNet
• Learning
28Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

29. TreNet
• LSTM
• Feed the sequence of
• Output
29Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

30. TreNet
• CNN
• Output
30Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

31. TreNet
• Feature Fusion and output layers
31Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

32. TreNet
• Learning
• Gradient descent
32Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

33. Experiments
• Datasets
• Daily Household Power Consumption (HousePC)
• Gas Sensor (GasSensor)
• Stock Transaction (Stock)
33
E Keogh, et al. “An online algorithm for segmenting time series”. ICDM, 2001
Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

34. Experiments
• Baselines
• CNN
• LSTM
• Support Vector Regression (SVR)
• Radial Basis kernel (SVRBF)
• Polynomial kernel (SVPOLY)
• Sigmoid kernel (SVSIG)
• Pattern-based Hidden Markov Model (pHMM)
34Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
P. Wang, et al. “Finding Semantics in Time Series”, SIGMOD 2011

35. Experiments
• Results: overall accuracy
35Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

36. Experiments
• Results: prediction visualization
36Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

37. Conclusion
37

38. Conclusion
38
• Hybrid neural networks
• TreNet for the Local Trend Learning
• Future work: a generic idea
• Social media streams
• Heterogeneous data
• Influence analysis
• And more…

39. Thanks! Q & A
This work is supported by EU OpenIoT Project
(Open Source Solution for the Internet of Things)
http://www.openiot.eu/
Feel free to contact: tian.guo0980@gmail.com, tian.guo@epfl.ch