BigData-JAWS 勉強会#9 アクセスログ収集〜分析基盤をAWSに構築しました。堅牢性、コスト、リアルタイム性をバランスする設計と、データ利用を促進する施策について紹介します。

1. AW S
Hajime Sano, Marketing & Data Technologist
Data Analytics & CRM Center, B to C Unit, Nikkei Inc.

2. 1.
2.
3. TIPS

4. •
• AWS
DynamoDB DAX
•
• www.linkedin.com/in/hsano

5. 1.
2.
3. ≠

6. 1.
2.
3. TIPS

7. +

9. frequency √ volume

10. 1 2 3 4 5 6 7 8 9 10 11 12 13 14
= LOG2(F√V)

11. • 10
•
•
•
• FT …

12. • GROUP BY 50
•
• Latency
•
•

13. 2014 2015 2016 2017 2018

14. •
•
•
•

15. AT L A S …
•
•
•
• 10 1
•

16. 4

17. R D B
H A D O O P
B I
D A S H B O A R D

19. R D B
H A D O O P
B I
D A S H B O A R D

20. 1.
2.
3. TIPS

21. EndpointTracking Enrichment Consumers S3 AzkabanSQS Kinesis S3 Redshift
Consumers ESS3
ParserAdobe Analytics
Dynamo DB Dynamo DB
S3
Kinesis→ES
DataFeed
Kinesis→S3 S3→RS
E B / E C 2
Rundeck

22. Kinesis S3 Redshift
Athena
Spectrum
ES Kibana
Analytics
Firehose
Quick Sight
(R)
Jupyter
(py)
OSS BI/DS
B I

23. AW S
SQS Kinesis S3 Redshift

24. S Q S
•
-
-
-

25. K I N E S I S
• Kinesis Stream
-
- 7
-
• Firehose Analytics

26. S 3
•
• Redshift
• S3
- Redshift Athena Spectrum
-

27. R E D S H I F T
•
• postgres SQL
- BI
•
-

28. E L A S T I C S E A R C H E C 2
• Elasticsearch Service EC2
- c4.8xlarge x 3
- r4.xlarge x 1
•
- X-Pack Graph ML
- ES OS JVM

29. • Kinesis
- 200ms
- Consumer
• Elasticsearch
-
- INDEX 1
• Redshift
- 15 20

30. 10 20
10 1

31. …
AT L A S
A
G

32. 1.
2.
3. TIPS

33. •
• Lightning Talk
• SQL Data Dojo
•
•
• / IF

34. R Studio Server Shiny Anaconda(JupyterHub)
Chartio
Re:dash Kibana
DOMO
DataSquad Maia
KPI Screens

35. • 2
1.
2.
•
- Slack
-

37. J O I N - L E S S
• OUTER JOIN
-
- JOIN
• 1
- 1
- SELECT

38. J O I N - L E S S
GET URL URL URL

39. •
- time sliced table UNION ALL
- timestamptz
- WHERE
•
- 3 30 …
-

40. last 7 days 7

41. •
- Redash
…
-
•
- Redash
-
KILL

43. T I P S
1.
2.
3.
4.

44. T H A N K Y O U !

45. Data Technologist Data Scientist

46. Q U E S T I O N S ?