How to build a no data loss pipeline using Apache kafka

1. No Data Loss Pipeline
with Apache Kafka
Jiangjie (Becket) Qin @ LinkedIn

2. ● Data loss
o producer.send(record) is called but record
did not end up in consumer as expected
● Message reordering
o send(record1) is called before send(record2)
o record2 shows in broker before record1 does
o matters in cases like DB replication
Data loss and message reordering

3. Kafka based data pipeline
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
Today’s Agenda:
● No data loss
● No message reordering
● Mirror maker enhancement
○ Customized consumer rebalance listener
○ Message handler

4. Synchronous send is safe but slow...
producer.send(record).get()
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

5. Using asynchronous send with callback can
be tricky
producer.send(record,callback)
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

6. Producer can cause data loss when
● block.on.buffer.full=false
● retries are exhausted
● sending message without using acks=all
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

7. Is this good enough?
producer.send(record,callback)
● block.on.buffer.full=TRUE
● retries=Long.MAX_VALUE
● acks=all
● resend in callback when message send failed
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

8. Message reordering might happen if:
● max.in.flight.requests.per.connection > 1, AND
● retries are enabled
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
Kafka BrokerProducer
message 0
message 1
message 0 failed
retry message 0
Timeline

9. Message reordering might also happen if:
● producer is closed carelessly
o close producer in user thread, or
o close without using close(0)
Producer
Record
Accumulator
Sender Thread
Kafka Broker
Timeline
1.msg 0
2.callback(msg 0) ack expt.
User
Thread
close prod.
3.msg 1
notify

10. ● close producer in the callback on error
● close producer with close(0) to prevent further
sending after previous message send failed
Producer
Record
Accumulator
Sender Thread
Kafka Broker
Timeline
1.msg 0
2.callback(msg 0) ack expt.
User
Thread
close(0)
notify

11. To prevent data loss:
● block.on.buffer.full=TRUE
● retries=Long.MAX_VALUE (for some use cases)
● acks=all
To prevent reordering:
● max.in.flight.requests.per.connection=1
● close producer in callback with close(0) on send failure
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

12. Not a perfect solution:
● Producer needs to be closed to guarantee message
order. E.g. In mirror maker, one message send failure
to a topic should not affect the whole pipeline.
● When producer is down, message in buffer will still be
lost
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

13. Correct producer setting is not enough
● acks=all still can lose data when unclean
leader election happens.
● Two replicas are needed at any time to
guarantee data persistence.
Kafka Brokers
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

14. ● replication factor >= 3
● min.isr = 2
● Replication factor > min.isr
o If replication factor = min.isr, partition will
be offline when one replica is down
Kafka Brokers
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

15. Settings we use:
● replication factor = 3
● min.isr = 2
● unclean leader election disabled
Kafka Brokers
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

16. ● Consumer might lose message when offsets are
committed carelessly. E.g. commit offsets before
processing messages completely
o Disable auto.offset.commit
o Commit offsets only after the messages are
processed
Consumer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

17. Kafka based data pipeline
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
Today’s Agenda:
● No data loss
● No message reordering
● Mirror maker enhancement
○ Customized consumer rebalance listener
○ Message handler

18. ● Consume-then-produce pattern
● Only commit consumer offsets of
messages acked by target cluster
● Default to no-data-loss and no-
reordering settings
Mirror Maker Enhancement
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

19. ● Customized Consumer Rebalance
Listener
o Can be used to propagate topic change from
source cluster to target cluster. E.g.
partition number change, new topic
creation.
Mirror Maker Enhancement
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

20. ● Customized Message Handler, useful for
o partition-to-partition mirror
o filtering out messages
o message format conversion
o other simple message processing
Mirror Maker Enhancement
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

21. ● startup/shutdown acceleration
o parallelized startup and shutdown
o 26 nodes cluster with 4 consumer each
takes about 1 min to startup and shutdown
Mirror Maker Enhancement
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker

22. Q&A