More Related Content Similar to Kafka Connect by Datio (20) More from Datio Big Data (13) Kafka Connect by Datio7. Kafka Goals
Frontend
Server
Metrics
Server
Metrics UI Log Search
✓ Decouple data pipelines
✓ Provide persistence for
message data to allow
multiple consumers
✓ Optimize for high
throughput of messages
✓ Allow for horizontal scaling
of the system to grow as the
data stream grow
Database
Server Shopping
Cart
Backend
Server
15. 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12
Consumer
0
Consumer
1
Consumer
2
Consumer
Group
Partition 0
Partition 1
Partition 2
Partition 3
Partition 0
Partition 1
Partition 2
Partition 3
Partition 0
Partition 1
Partition 2
Partition 3
Partition 0
Partition 1
Partition 2
Partition 3
Consumer 1
Consumer 2
Consumer 1
Consumer 2
Consumer 3
Consumer 4
Consumer 1
Consumer 2
Consumer 3
Consumer 4
Consumer 5
Consumer 6
Topic T1 Topic T1 Topic T1Consumer Group 1 Consumer Group 1
Consumer
Group 1
Consumer
21. Producer
Record
Serializer
Partitioner
Topic A
Partition 0
Batch 0
Batch 1
Topic B
Partition 1
Batch 0
Batch 1
Fail
?
Retry
?
Yes
Yes
MetadataException
Send()Topic
Partition
Key
Value
Topic
Partition
commit
Metadata
Topic
Partition
Offset
Broker
Worker settings to ensure no data loss
request.timeout.ms=MAX_VALUE
retries=MAX_VALUE
max.in.flight.request.per.connection=1
acks=all
max.block.ms=MAX_VALUE
22. Worker 2 Worker 3Worker 1
Worker
Task
Worker
Task
Worker
Task
Worker
config
Source
config
Source
config
Source
config
Worker
config
Worker
Task
Worker
Task
Worker
Task
Conn 1, Task 3
Partitions: 5,6
Conn 2, Task 1
Partitions: 1,2
Conn 2
Conn 1, Task 2
Partitions: 3,4
Conn 1
Conn 1, Task 1
Partitions: 1,2
Conn 2, Task 2
Partitions: 3,4
Standalone
worker
Scalability
Fault tolerance
Share
connectors &
tasks
Distributed Worker
Simple
1 Worker
N conn/tasks