2015/01/29 RWDA final presentation

1. Prediction of Exchange Rate
Using Deep Neural Network
名古屋大学 情報科学研究科
武田研究室
林 知樹
1

2. Agenda
1. Background
2. Outline of Deep Learning
3. Proposed method
 The structure of proposed model
 features
4. Experiments
5. Conclusion
2

3. Background
Earning without working is
a dream for humans.
3
What is the shortest way
to achieve this?
The answer is FX

4. Background
 FX is money exchange game.
 The shortest way to achieve our dream.
 How to win?
 This is very simple.
4
All you have to do is
only predicting up or down.
When the value is high, sell.
When the value is low, buy.

5. How to predict
 My Hypothesis is
 Prediction Using Deep Neural Network : DNN
 State-of-the-art machine learning method
5
Future exchange rate
consists of past information.

6. Deep Neural Network (DNN)
6
Input
layer
Middle
layers
Output
layer
Layered Neural Network has a lot of middle layers
Deep Neural Network ：
In general,
#middle layer > 3
The difference is only #middle layer.

7. Deep Neural Network (DNN)
 The structure of DNN doesn’t look new
 We can’t train DNN with conventional method.
 Initial parameters : randomization
→ Fall into bad local solution
 Appropriate initialization method appeared.
 Pre-training by RBM or Auto-Encoder
7
We can prevent the
disappearance of gradient.
but
Disappearance of gradient problem

8. EX. Image recognition
 Before appearance of DNN
 Appearance of DNN
8
Raw data Vector expression
Feature
Extraction
Discriminator
Recognition
result
Raw data
Feature
Extraction Recognition
Deep Learning
Human-made
Training~
Learning comprehensively
from feature extraction to discriminative system

9. EX. Image recognition
 Before appearance of DNN
 Appearance of DNN
9
Raw data Vector expression
Feature
Extraction
Discriminator
Recognition
result
Raw data
Feature
Extraction Recognition
Deep Learning
Human-made
Training~
Learning comprehensively
from feature extraction to discriminative system

10. EX. Image recognition
 Before appearance of DNN
 Appearance of DNN
10
Raw data Vector expression
Feature
Extraction
Discriminator
Recognition
result
Raw data
Feature
Extraction Recognition
Deep Learning
Human-made
Training~
Learning comprehensively
from feature extraction to discriminative system
Achieved highest score
using only raw data.

11. Proposed method
 2 kind of approach
1. Direct prediction of the exchange rate
 Like Regression
 Next time value is used as supervised data.
2. Binary option
 2 Class Classification problem
• In next time, the value become high → Class 1
• In next time, the value become low → Class 0
 I used these value as supervised data.
11

12. Regression by DNN
12
Middle layer
Output layer
xxh )(
)( bhy  Wz
y
z
Regression→no range
Output identity mapping
Output
Real value
Identity function
W

13. 2 Class Classification by DNN
13
]1,0[
)exp(1
1
)(
x
xh


z
2 class → 0 or 1
Output is prob.→ [ 0, 1 ]
Sigmoid function
y
WMiddle layer
Output layer
)( bhy  Wz
Output

14. Input Features
 We used 10 kind of features as inputs.
 Raw value
 Exchange value
 Top price
 Low price
 Closing value
 Moving Average (9 points)
 Relative Strength Index (RSI)
 Stochastics RSI
 Slow stochastics
 Fast stochastics
 Williams %R
14
Total 10 dim.

15. DNN input
15
𝐷 dim. feature
・・・
𝐷 dim.
N frame
DNN can deal with high dimension features and many frames.
time
Total (𝑁 + 1) × 𝐷 dim.
Concatenated feature

16. Flowchart
16
Calculatingfeatures
Rawvalues
10dim.feature
Featureconcatenation
TrainingDNNusingfeatures
Training
phase
Testing
phase
100dim.feature
TrainedDNN
Inputconcatenatedfeature
fortesting
Predictedvalue

17. Flowchart
17
Calculatingfeatures
Rawvalues
10dim.feature
Featureconcatenation
TrainingDNNusingfeatures
Training
phase
Testing
phase
100dim.feature
TrainedDNN
Inputconcatenatedfeature
fortesting
Predictedvalue
 Exchange value
 Top price
 Low price
 Closing value

18. Flowchart
18
Calculatingfeatures
Rawvalues
10dim.feature
Featureconcatenation
TrainingDNNusingfeatures
Training
phase
Testing
phase
100dim.feature
TrainedDNN
Inputconcatenatedfeature
fortesting
Predictedvalue
 Raw values
 Moving Average
 RSI
 Stochastics RSI
 Williams %R

19. Flowchart
19
Calculatingfeatures
Rawvalues
10dim.feature
Featureconcatenation
TrainingDNNusingfeatures
Training
phase
Testing
phase
100dim.feature
TrainedDNN
Inputconcatenatedfeature
fortesting
Predictedvalue
time

20. Flowchart
20
Calculatingfeatures
Rawvalues
10dim.feature
Featureconcatenation
TrainingDNNusingfeatures
Testing
phase
100dim.feature
TrainedDNN
Inputconcatenatedfeature
fortesting
Predictedvalue
1. Pre-training
 RBM
2. Fine-tuning
 back propagation
Training
phase

21. Flowchart
21
Calculatingfeatures
Rawvalues
10dim.feature
Featureconcatenation
TrainingDNNusingfeatures
Training
phase
100dim.feature
TrainedDNN
Inputconcatenatedfeature
fortesting
Predictedvalue
Predicted value
100 dim. feature
Test data
Testing
phase

22. An Exchange rate data
22
 including data from 1991 to 2014.
 Time interval is 1 hour.
Date Time
Exchange
rate
Top
price
Low
price
Closing
value
Total
transaction

23. Experiment
 Experimental conditions
 DNN training conditions
23
Dataset $-¥ Exchange rate
# data
1991/01/04 ~ 2015/01/5
Total 97362 points
# DNN layer 5 layers
# middle layer node 256 nodes
Pre-training Fine-tuning
Learning rate 0.002 0.00006
Momentum 0.9 0
Batch size 128 128
Epoch 30 50

24. Dividing dataset
24
Training data ①
Training data ②
Training data ③
Test data ①
Test data ②
Test data ③
 Each test data has 24 points(24 hours).
 In this time, I made from ① to ㉛.

25. Direct prediction
 Input :
 Features calculated by presence and past signal
 Output :
 the next time closing value
26

26. Direct prediction result
27

27. Direct prediction result
28
Closed test
Close up

28. Direct prediction result
29
Closed test
Prediction could capture characteristics of answer line.

29. Direct prediction result
30
Open test
Close up

30. Direct prediction result
31
Open test
Predicted signal fluctuates.
There is no information about that
in the next time the value will become up or down.

31. Direct prediction result
32
Open test
Predicted signal fluctuates.
There is no information about that
in the next time the value will become up or down.

32. Binary option
 Input :
 Features calculated by presence and past signal
 Output :
 In the next time, up(Class 1) or down(Class 0)
33

33. Binary option result
 Closed test
 Open test
34
)96366/51516([%]46.53Acc.
)744/375([%]40.50Acc.
Using dice is better than this method.

34. Using dice is better than this method.
Binary option result
 Closed test
 Open test
35
)96366/51516([%]46.53Acc.
)500/252([%]40.50Acc.

35. Why we couldn’t predict?
 Small fluctuation prevents us from predicting.
36

36. Why we couldn’t predict?
 Small fluctuation prevents us from predicting.
37
Similar to white noise

37. Why we couldn’t predict?
 Small fluctuation prevents us from predicting.
38
Similar to white noise

38. Another approach
 Prediction of trend transition
 Trend transition means
 The value will become up or down
for Moving average in the past 𝑁 hours.
 We can ignore the effect of small fluctuation.
39

39. Prediction of trend transition
40
 Input :
 Features calculated by presence and past signal
 Output :
 In the next time, trend will become up(Class 1)
or down(Class 0)

40. Prediction of trend transition
 Closed test
 Open test
41
)97338/81516([%]75.83Acc.
)744/678([%]63.87Acc.

41. 42
Prediction of trend transition
Predicted value is [0,1].
The closer to 1 or 0 predicted value is,
the more reliable the prediction is.
We can set the threshold
to make the prediction more reliable.
 Open test (Setting Threshold as 0.8 and 0.2)
)515/487([%]61.94Acc.

42. 43
Prediction of trend transition
Predicted value is [0,1].
The closer to 1 or 0 predicted value is,
the more reliable the prediction is.
We can set the threshold
to make the prediction more reliable.
 Open test (Setting Threshold as 0.8 and 0.2)
)515/487([%]61.94Acc.

43. Conclusion and future works
 Conclusion
 We try to predict exchange rate using DNN.
 3 kind of approach
 Direct prediction
 Binary option
 Trend transition
 We could predict trend transition
with 83%(Closed) and 87%(Open).
 Future problem
 Considering another kind of feature
 Prediction of more long term change
44
Failed…
Failed…
Succeeded!!

44. Thank you for your attention!
45

45. Pre training
 RBMによる貪欲学習
 1層目と2層目をRBMとみなして学習
 2層目の出力をサンプリング
 2層目と3層目をRBMとみなして学習
 以下繰り返し
(1)1段目のRBM学習 (2)2段目のRBM学習 (3)3段目のRBM学習