Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Awsでつくるapache kafkaといろんな悩み
Next
Download to read offline and view in fullscreen.

Share

Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)

Download to read offline

D&S Data Night vol.04

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)

  1. 1. Apache Kafka & Kafka Connect ( ETL ) Future Architect ,Inc Keigo Suda 2016/11/25 D&S Data Night vol.04
  2. 2. 12 ! Kafka Connect ETL ( ) ( ) /
  3. 3. 13 ! Future Architect ,Inc ( ) ! ! IoT
  4. 4. 14
  5. 5. 15 Kafka Connect
  6. 6. 16 ! Kafka ver 0.9 ! Kafka ! ! ! / Kafka Connect
  7. 7. 17 Connectors https://www.confluent.io/product/connectors/
  8. 8. Kafka Connect
  9. 9. 21
  10. 10. 22 ! ! ! Kafka Connect ( )
  11. 11. 23 ETL Kakfa Connect( ) P T // / / AC KM I M KM
  12. 12. Kafka Connect
  13. 13. 25http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started Kakfa Connect Kafka Connect
  14. 14. 26 Kakfa Connect http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
  15. 15. 27 Stream & Partition(RDB ) http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
  16. 16. 28 N:1 Stream & Partition(RDB ) http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
  17. 17. 29 Kakfa Connect http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
  18. 18. 30 Worker & Connector http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
  19. 19. 31 CONNECTER WORKER STREAM PARTITION STANDALONE MODE DISTRIBUTED MODE TASK
  20. 20. 32 $ bin/connect-standalone.sh config/connect-standalone.properties connector1.properties (Standalone mode) ! ! ! Worker ! Connector Worker Connector
  21. 21. 33 name=local-file-source connector.class=FileStreamSource tasks.max=1 file=test.txt topic=connect-test bootstrap.servers=localhost:9092 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter ... Worker Connector
  22. 22. 35 ! ! ! ETL
  23. 23. 36 ! Kafka Connect Source Sink ! Sink/Source Sink Kafka Connect Source Sink
  24. 24. 37 ! Connector/Task OK ! API OK ( ) ( )
  25. 25. 38 Connector( )
  26. 26. 39 !!(source pull) Task
  27. 27. 40 ! Task Connector ! Deploy name=sample-sink connector.class=SampleSinkConnector tasks.max=1 topic=connect-test Connector
  28. 28. 42 ! Kafka ! Kafka Connect
  29. 29. 43 ! API Kafka Connect
  30. 30. 44 (kafka-connect-jdbc)
  31. 31. 45 ! Confluent Confuluent Platform Kafka Connect
  32. 32. 46 ! Kakfa ( ) ! Confluent….
  • nowavailable

    Oct. 21, 2019
  • TakumiYamaoka

    Oct. 10, 2019
  • SachihoWakita

    Nov. 16, 2018
  • HirofumiKawasaki

    Jun. 13, 2018
  • KenTakehara

    Aug. 18, 2017
  • takushimizu

    Jul. 28, 2017
  • toutoutakataka

    Jun. 12, 2017
  • YasuhiroMurata

    Jun. 7, 2017
  • kkitase

    May. 20, 2017

D&S Data Night vol.04

Views

Total views

3,852

On Slideshare

0

From embeds

0

Number of embeds

156

Actions

Downloads

21

Shares

0

Comments

0

Likes

9

×