1. Time Series analysis
by JavaScript
LL matsuri 2013/08/24
1

2. Time Series analysis
by JavaScript
前処理
LL matsuri 2013/08/24
2

3. {twitter: “muddydixon”}
✓Data Mining
✓Data Visualization
✓Love D3.js
✓working @ NIFTY
3

4. NIFTY Cloud
4

5. Visualization
✓What is Visualization?
communicating information
clearly and effectively
5

6. Visualization on Web
✓Profit
✓Watch KGI / KPI continuously
✓Share visualization via URL
✓empowered representation by HTML5 / css3
-> D3.js
6

7. Visualization on Web : D3.js
7

8. Time Series Visualization
✓Visualize “time series” of
✓KGI / KP of services
✓Metrics of system resources
8

9. Sample
{"time":"2013-01-02T08:54:31.000Z","cpu":0.6283,"memory":0.4427}
{"time":"2013-01-02T08:54:22.000Z","cpu":0.6103,"memory":0.6494}
{"time":"2013-01-02T08:54:24.000Z","cpu":0.6096,"memory":0.4888}
{"time":"2013-01-02T08:54:25.000Z","cpu":0.6811,"memory":0.641}
{"time":"2013-01-02T08:54:14.000Z","cpu":0.6028,"memory":0.6452}
{"time":"2013-01-02T08:54:06.000Z","cpu":0.5565,"memory":0.4346}
{"time":"2013-01-02T08:54:10.000Z","cpu":0.4161,"memory":0.358}
{"time":"2013-01-02T08:55:01.000Z","cpu":0.5256,"memory":0.5969}
{"time":"2013-01-02T08:55:47.000Z","cpu":0.6145,"memory":0.3213}
{"time":"2013-01-02T08:56:24.000Z","cpu":0.6896,"memory":0.5109}
{"time":"2013-01-02T08:56:35.000Z","cpu":0.443,"memory":0.633}
{"time":"2013-01-02T08:56:11.000Z","cpu":0.4746,"memory":0.4777}
{"time":"2013-01-02T08:56:42.000Z","cpu":0.503,"memory":0.5732}
{"time":"2013-01-02T08:56:38.000Z","cpu":0.4731,"memory":0.4024}
{"time":"2013-01-02T08:57:01.000Z","cpu":0.405,"memory":0.5203}
{"time":"2013-01-02T08:57:54.000Z","cpu":0.4496,"memory":0.5324}
{"time":"2013-01-02T08:57:04.000Z","cpu":0.3528,"memory":0.3145}
{"time":"2013-01-02T08:57:49.000Z","cpu":0.6659,"memory":0.4285}
{"time":"2013-01-02T08:57:19.000Z","cpu":0.3237,"memory":0.6979}
{"time":"2013-01-02T08:58:31.000Z","cpu":0.4045,"memory":0.6581}
{"time":"2013-01-02T08:58:45.000Z","cpu":0.3134,"memory":0.3261}
{"time":"2013-01-02T08:58:44.000Z","cpu":0.4382,"memory":0.419}
{"time":"2013-01-02T08:58:38.000Z","cpu":0.6983,"memory":0.3493}
{"time":"2013-01-02T08:58:34.000Z","cpu":0.6645,"memory":0.6272}
{"time":"2013-01-02T08:59:54.000Z","cpu":0.604,"memory":0.3287}
{"time":"2013-01-02T08:59:25.000Z","cpu":0.574,"memory":0.4856}
{"time":"2013-01-02T08:59:19.000Z","cpu":0.4775,"memory":0.3998}
{"time":"2013-01-02T08:59:34.000Z","cpu":0.5047,"memory":0.6702}
{"time":"2013-01-02T08:59:46.000Z","cpu":0.672,"memory":0.5021}
{"time":"2013-01-02T08:59:53.000Z","cpu":0.3278,"memory":0.6679}
{"time":"2013-01-02T08:59:29.000Z","cpu":0.4494,"memory":0.3319}
9

10. 可視化エンジニア
マネージャー
デイリーで
売上データ見せて
了解しました
10

11. 可視化エンジニア
マネージャー
やっぱり、月次
と週次で
…了解しました
11

12. 可視化エンジニア
マネージャー
あと売上の平均
と分散も
うっ、つらい
12

13. Summarize
keyvalues = {}
for d in data
time = new Date(d.time.getFullYear(), d.time.getMonth(),
d.time.getDate(), d.time.getHours(), d.time.getMinutes())
keyvalues[time] = [] unless keyvalues[time]
keyvalues[time].push d
aggregated = []
for time, values of keyvalues
obj = {time: time, cpu: 0, memory: 0}
for d in values
obj.cpu += d.cpu
obj.memory += d.memory
aggregated.push obj
console.log aggregated
13

14. Problems
Large
Time perspectives / Statistics
Avoid to get data each time
Hourly / Daily / Weekly / Monthly / Yearly
Summary / Average / Stdev
14

15. Problems
メンドクサイ
15

16. Series.js
16

17. Series.js
✓Providing utility methods for time series
✓sum, sum square, mean, median, quantile
✓variance, standard devience, covariance, correlation
✓auto covariance, auto correlation
✓Providing aggregation
✓minutes, hour, day, week, month, year
✓Coming soon
✓auto regression, spectrum
✓moving average, holt-winters, exponential moving average
17

18. Series.js APIs example
# class methods
Series.sum([1,2,3,4,5]) #=> 15
Series.mean([1,2,3,4,5]) #=> 3
# instance methods
s = new Series()
s.put d for d in [1,2,3,4,5]
s.mean() #=> 3
18

19. Series.js APIs example
# accessor
Series.y((d)-> d.v).sum([{v:1},{v:2},{v:3},{v:4},{v:
5}]) #=> 15
s = new Series().y((d)-> d.v)
s.put d for d in [{v:1}, {v:2}, {v:3}, {v:4}, {v:5}]
s.mean() #=> 3
19

20. Series.js APIs example
# aggregation
data = [{t: 1, v:1}, {t: 1, v:2}, {t:2, v:3}, {t:2, v:4},
{t:1, v:5}]
Series.y((d)-> d.v).aggregation(Series.sum).key((d)-
> d.t)(data)
#=> [{t:1, v: 8}, {t: 2, v: 7}]
20

21. Sample
{"time":"2013-01-02T08:54:31.000Z","cpu":0.6283,"memory":0.4427}
{"time":"2013-01-02T08:54:22.000Z","cpu":0.6103,"memory":0.6494}
{"time":"2013-01-02T08:54:24.000Z","cpu":0.6096,"memory":0.4888}
{"time":"2013-01-02T08:54:25.000Z","cpu":0.6811,"memory":0.641}
{"time":"2013-01-02T08:54:14.000Z","cpu":0.6028,"memory":0.6452}
{"time":"2013-01-02T08:54:06.000Z","cpu":0.5565,"memory":0.4346}
{"time":"2013-01-02T08:54:10.000Z","cpu":0.4161,"memory":0.358}
{"time":"2013-01-02T08:55:01.000Z","cpu":0.5256,"memory":0.5969}
{"time":"2013-01-02T08:55:47.000Z","cpu":0.6145,"memory":0.3213}
{"time":"2013-01-02T08:56:24.000Z","cpu":0.6896,"memory":0.5109}
{"time":"2013-01-02T08:56:35.000Z","cpu":0.443,"memory":0.633}
{"time":"2013-01-02T08:56:11.000Z","cpu":0.4746,"memory":0.4777}
{"time":"2013-01-02T08:56:42.000Z","cpu":0.503,"memory":0.5732}
{"time":"2013-01-02T08:56:38.000Z","cpu":0.4731,"memory":0.4024}
{"time":"2013-01-02T08:57:01.000Z","cpu":0.405,"memory":0.5203}
{"time":"2013-01-02T08:57:54.000Z","cpu":0.4496,"memory":0.5324}
{"time":"2013-01-02T08:57:04.000Z","cpu":0.3528,"memory":0.3145}
{"time":"2013-01-02T08:57:49.000Z","cpu":0.6659,"memory":0.4285}
{"time":"2013-01-02T08:57:19.000Z","cpu":0.3237,"memory":0.6979}
{"time":"2013-01-02T08:58:31.000Z","cpu":0.4045,"memory":0.6581}
{"time":"2013-01-02T08:58:45.000Z","cpu":0.3134,"memory":0.3261}
{"time":"2013-01-02T08:58:44.000Z","cpu":0.4382,"memory":0.419}
{"time":"2013-01-02T08:58:38.000Z","cpu":0.6983,"memory":0.3493}
{"time":"2013-01-02T08:58:34.000Z","cpu":0.6645,"memory":0.6272}
{"time":"2013-01-02T08:59:54.000Z","cpu":0.604,"memory":0.3287}
{"time":"2013-01-02T08:59:25.000Z","cpu":0.574,"memory":0.4856}
{"time":"2013-01-02T08:59:19.000Z","cpu":0.4775,"memory":0.3998}
{"time":"2013-01-02T08:59:34.000Z","cpu":0.5047,"memory":0.6702}
{"time":"2013-01-02T08:59:46.000Z","cpu":0.672,"memory":0.5021}
{"time":"2013-01-02T08:59:53.000Z","cpu":0.3278,"memory":0.6679}
{"time":"2013-01-02T08:59:29.000Z","cpu":0.4494,"memory":0.3319}
21

22. Summarize with Series.js
Series.y((d)-> d.cpu).aggregation(Series.sum)
.minute((d)-> d.time)(data)
#=> [{ t: 'Sun Jan 13 2013 23:29:00 GMT+0900 (JST)', y: 2.0095 },
{ t: 'Sun Jan 13 2013 23:30:00 GMT+0900 (JST)', y: 3.4370 },
{ t: 'Sun Jan 13 2013 23:31:00 GMT+0900 (JST)', y: 1.58180 },
{ t: 'Sun Jan 13 2013 23:32:00 GMT+0900 (JST)', y: 2.2742 },
{ t: 'Sun Jan 13 2013 23:33:00 GMT+0900 (JST)', y: 3.4541 },
{ t: 'Sun Jan 13 2013 23:34:00 GMT+0900 (JST)', y: 4.6035 },
{ t: 'Sun Jan 13 2013 23:35:00 GMT+0900 (JST)', y: 2.1165 },
{ t: 'Sun Jan 13 2013 23:36:00 GMT+0900 (JST)', y: 1.6177 },
22

23. TODO
Performance
Refactoring
Document
23

24. TODO
Client side
24

25. TODO
Client side
さっきpushしました
25

26. CROSS 2014
26

27. 以下、ビールを飲んで
笑顔な技術者の顔が
続いていると
思ってください
27