ビッグデータのリアルタイム処理技術勉強会 http://futureofdata.connpass.com/event/40077/ 発表資料

1. / @laclefyoshi / ysaeki@r.recruit.co.jp

2. •
•
•
•
•
•
2

3. • 2011/04
• 2015/09
•
• Druid (KDP, 2015)
• RDB NoSQL ( , 2016; : HBase )
• ESP8266 Wi-Fi IoT (KDP, 2016)
•
• (DEIM 2014)
• (WebDB Forum 2014)
• Spark Streaming (Spark Meetup December 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• Kinesis Family (R-JAWS Bigdata; 2016)
3

5. 5

7. • Queue Queueing system
•
• Message
• 1
• Broker
•
• Producer
•
• Consumer
•
• Protocol
•
7

8. •
•
•
8

9. 9
Apache ActiveMQ
Apache Kafka
RabbitMQ
NATS
NSQ
Redis
ZeroMQ
Nanomsg

10. Apache Kafka
A distributed streaming platform
10

11. Apache ActiveMQ
The most popular and powerful open source
messaging and Integration Patterns server
11

12. RabbitMQ
Robust messaging for applications
Easy to use
12

13. NATS
For cloud native applications, IoT messaging,
and microservices architectures
13

14. NSQ
A realtime distributed messaging platform
14

15. Redis
In-memory data structure store,
used as database, cache and message broker
15

16. ZeroMQ
An embeddable networking library
but acts like a concurrency framework
16

17. Nanomsg
A socket library that provides
several common communication patterns
17

19. 19

20. 20

21. :
•
•
• Lambda Architecture
:
•
•
21

22. 22

23. :
•
•
:
•
23

24. 24

25. :
•
•
:
•
25

27. 27

28. : ……
https://cloud.google.com/blog/big-data/2016/09/
apache-kafka-for-gcp-users-connectors-for-pubsub-dataflow-and-bigquery
28
Pub/Sub
Dataflow/Apache Beam
BigQuery
Apache Kafka

29. 29

30. 30
:
•
:
•

31. 31

32. 32
:
•
:
•

33.
2016/10/25
official / 3rd party
33

34. • JIRA Github
• Apache Flink Apache Spark
Apache Bahir (3rd party)
• Apache Kafka
34

36. 1000
• [ 1] → [ 1 → 1]
• AWS: EC2 /t2.medium x 2 ( VPC)
• github:laclefyoshi/queues-benchmark
• Docker
• Python
36

37. 1000
37
•
•
•
• 1000 Byte 100000 Byte
1000 Byte
• 1000

38. 38

40. 40

41. 41

42. Queue makes everyone happy!!