Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Kafka AWS Kinesis
/ @laclefyoshi / ysaeki@r.recruit.co.jp
•
• Kafka Kinesis
•
•
•
•
2
• 2011/04
• 2015/09
•
• Druid (2015; Kindle Direct Publishing)
• RDB NoSQL (2016; ; HBase )
•
• (WebDB Forum 2014)
• Spark...
Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Fireho...
Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Fireho...
:
5
: <100ms
: ~8000msgs/sec
Kinesis
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL 7
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL
OSS
•Kafka Streams
•Pipeline...
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[ ]
http://insightdataengineering.com/blog/ingestion-comparison/
X
Kafka Kinesis
• Kafka or Kinesis
• Kafka OSS
Kinesis
•
9
Kafka Kinesis
• (< 5000msgs/sec) Kinesis
• Kafka
• Kinesis
• Hadoop Kafka
• Kinesis
• Kafka
10
• OSS → Kafka
• → Kinesis
• API
•
• SNS Bot IoT
•
11
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
Kafkaを使った マイクロサービス基盤 part2 +運用して起きたトラブル集
Next
Download to read offline and view in fullscreen.

27

Share

Download to read offline

KafkaとAWS Kinesisの比較

Download to read offline

Apache Kafka Meetup Japan #1 http://connpass.com/event/32293/ LT発表資料

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

KafkaとAWS Kinesisの比較

  1. 1. Kafka AWS Kinesis / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • • Kafka Kinesis • • • • 2
  3. 3. • 2011/04 • 2015/09 • • Druid (2015; Kindle Direct Publishing) • RDB NoSQL (2016; ; HBase ) • • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) 3
  4. 4. Kafka Kinesis • Apache • Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
  5. 5. Kafka Kinesis • Apache • Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
  6. 6. : 5 : <100ms : ~8000msgs/sec Kinesis
  7. 7. [Kafka] [Kinesis] 6
  8. 8. [Kafka] [Kinesis] 6
  9. 9. [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS API Gateway HTTP API ETL ETL 7
  10. 10. [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS API Gateway HTTP API ETL ETL OSS •Kafka Streams •PipelineDB AWS •Kinesis Analytics 7
  11. 11. [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ← → ← → ← → 8
  12. 12. [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ← → ← → ← → 8
  13. 13. [ ] http://insightdataengineering.com/blog/ingestion-comparison/ X
  14. 14. Kafka Kinesis • Kafka or Kinesis • Kafka OSS Kinesis • 9
  15. 15. Kafka Kinesis • (< 5000msgs/sec) Kinesis • Kafka • Kinesis • Hadoop Kafka • Kinesis • Kafka 10
  16. 16. • OSS → Kafka • → Kinesis • API • • SNS Bot IoT • 11
  • yoshioshinozaki7

    Apr. 14, 2020
  • t2y

    Jan. 6, 2020
  • Tadasihasimoto

    Apr. 26, 2019
  • Zengzhen2

    Nov. 23, 2018
  • KeisukeKadoyama

    Nov. 4, 2018
  • KoheiSenkawa

    Oct. 6, 2018
  • taroshun

    Aug. 21, 2018
  • tosikawa

    Jul. 23, 2018
  • maemichiyuya

    May. 24, 2018
  • reapasisow

    Mar. 1, 2018
  • KenichiTakahashi1

    Oct. 1, 2017
  • KensukeFujiya

    Jun. 1, 2017
  • ysugiura

    Mar. 20, 2017
  • keisukekosaki

    Mar. 15, 2017
  • wing3298

    Jan. 18, 2017
  • AoiroAoino

    Dec. 28, 2016
  • green_air

    Dec. 11, 2016
  • nm869

    Oct. 27, 2016
  • saicologic

    Oct. 26, 2016
  • mafuyukamono

    Oct. 24, 2016

Apache Kafka Meetup Japan #1 http://connpass.com/event/32293/ LT発表資料

Views

Total views

12,807

On Slideshare

0

From embeds

0

Number of embeds

3,751

Actions

Downloads

67

Shares

0

Comments

0

Likes

27

×