When working with KafkaConsumer, we usually employ single thread both for reading and processing of messages. KafkaConsumer is not thread-safe, so using single thread fits in well. Downside of this approach is that you are limited to single thread for processing messages. By decoupling consumption and processing, we can achieve processing parallelization with single consumer and get the most out of multi-core CPU architectures available today. While this can be very useful in certain use-case scenarios, it's not trivial to implement. How do we use multiple threads with KafkaConsumer which is not thread safe? How do we react to consumer group rebalancing? Can we get desired processing and ordering guarantees? In this talk we 'll try to answer these questions and explore challenges we face on our path.

1. Decoupling consumption and processing
for better resource utilization
Igor Buzatović
@buzzgor
beegor

2. Thread per consumer model
poll
★ getting records
★ processing records
★ committing offsets
★ handling consumer group rebalance
process
Single thread is:
If subscription model with default configuration is used,
everything on this list happens inside the poll method,
except processing

3. Key considerations
★ Offset committing
★ Group rebalancing
By default, both handled automatically
... when implementing threading model

4. Key considerations
★ Automatic
○ executed periodically
○ commit interval specified by auto.commit.interval.ms config (default value = 5000)
★ Manually
○ executed by calling commitSync/commitAsync method
○ commit offsets of the last processed record
Committing offsets

5. Key considerations
★ Each consumer group is coordinated by one of the brokers (group coordinator)
○ It manages group joining and leaving
○ One broker can be coordinator for many groups
★ One of the consumers in the group is the group leader
○ Responsible for partition assignment
★ Two versions of protocol
○ Eager rebalancing (old)
○ Incremental cooperative rebalancing (new)
Group rebalancing quick overview

6. Key considerations
★ Rebalancing is triggered by group coordinator when:
○ New consumer joins or existing consumer leaves the group
○ New partitions are added to a topic that is part of subscription
○ Consumer subscription changes
★ Coordinator notifies consumers about rebalance through heartbeat responses
★ Consumers must finish processing for revoked partitions before they can be reassigned
★ Consumers confirms partitions release by sending group join request on next poll
Group rebalancing quick overview

7. Motivation for multi-threaded model
Why don’t we just add more consumers ?
★ More consumers = more TCP connections to the cluster.
○ Kafka handles connections very efficiently so this is generally a small cost.
★ More consumers = more requests being sent to the server
○ causes slightly less batching of data which can cause some drop in I/O throughput.
★ Long record processing
○ can cause group rebalance due to poll interval timeout
○ can cause long rebalancing periods -waiting consumers to re-join group

8. Multi-threaded implementations
Disclaimer :)
Implementations show in this presentation are not meant to be production
ready.
The goal is to point out the challenges we face while implementing
multithreaded model and only suggest way to overcome them.
There are many more ways to implement multithreaded model.

9. “Fork join” threading model
poll process process process ...
★ Use multiple threads for processing
but wait for all of them to finish
before next poll call
★ To keep processing order
guaranties, process records from
same partition by same thread
★ Offset commiting and rebalancing
handled the same way as with single
thread model (by the poll method)

10. “Fork join” threading model
“Fork Join” demo implementation
1. Get records collection from poll method
2. Group records by partition (results in multiple collections of records)
3. Create Task (Callable implementation) for each collection from step 2
4. Use ExecutorService.invokeAll() to submit tasks from step 3
How it works

11. “Fork join” threading model
“FORK JOIN” DEMO

12. “Fully decoupled” multi-threaded model
poll submit tasks
process
process
process
★ Use multiple threads for processing
★ To keep processing order
guaranties, process records from
same partition by same thread
★ Do not wait for processing threads
to finish, instead continue to poll
★ Offset must be committed manually
★ Use ConsumerRebalanceListener
so you can finish processing for
revoked partitions

13. Decoupled multi-threaded model
★ Poll method is called in parallel with processing - automatic offset committing is not an option
★ Offsets must be committed manually in order to keep at-least-once delivery semantics
★ Thread synchronization is required in order to commit offsets only after processing is done
Committing offsets

14. Decoupled multi-threaded model
★ Records from an individual partition must be processed by only one thread at the time
★ This can be achieved by :
○ Pausing the partition after submitting its records for processing
○ Resuming the partition after records processing is finished
○ KafkaConsumer is not thread safe, so thread synchronization is required when resuming
Preserving processing order guarantees

15. Decoupled multi-threaded model
★ Consumer joins group rebalance during poll method execution
★ Some of the partitions assigned to the consumer can be revoked while its records are still being
processed by another thread
★ Registering ConsumerRebalanceListener gives us the ability to react to partition revocation
★ Upon receiving partition revoked event we can:
○ Do nothing - leads to unnecessary duplicate record processing and resource consumption
○ Wait for processing task to finish
○ Stop processing task (wait only current record processing to finish)
Reacting to group rebalance

16. Decoupled multi-threaded model
“Fully decoupled” demo implementation
1. Get records collection from poll method
2. Group records by partition (results in multiple collections of records)
3. Create Task (Runnable implementation) for each collection from step 2
4. Use ExecutorService.submit() to submit tasks from step 3
5. Pause partitions for submitted tasks
6. Store reference to submitted tasks so we can access them later
Main consumer thread

17. Decoupled multi-threaded model
“Fully decoupled” demo implementation
public void run() {
while (!stopped.get()) {
ConsumerRecords<String, String> records = consumer.poll(...);
handleFetchedRecords(records);
checkActiveTasks();
commitOffsets();
}
}
Main consumer thread

18. Decoupled multi-threaded model
“Fully decoupled” demo implementation
void handleFetchedRecords(ConsumerRecords<...> allRecords) {
records.partitions().forEach(partition -> {
List<ConsumerRecord> partitionRecords = allRecords.records(partition);
Task task = new Task(partitionRecords);
executor.submit(task);
activeTasks.put(partition, task);
});
consumer.pause(records.partitions());
}
Main consumer thread

19. Decoupled multi-threaded model
“Fully decoupled” demo implementation
1. Commit offsets from task processing thread
○ Requires thread synchronization
○ Can cause too frequent commits
2. Commit offsets from main thread
○ Ask task for current offset on each main loop iteration and store them to a buffer
○ Commit offsets from buffer periodically
Offset committing - two approaches

20. Decoupled multi-threaded model
“Fully decoupled” demo implementation
private void checkActiveTasks() {
...
activeTasks.forEach((partition, task) -> {
...
long offset = task.getCurrentOffset();
if (offset > 0)
offsetsToCommit.put(partition, new OffsetAndMetadata(offset));
});
...
}
Offset commiting - from the main thread

21. Decoupled multi-threaded model
“Fully decoupled” demo implementation
private void commitOffsets() {
try {
long currentTimeMillis = System.currentTimeMillis();
if (currentTimeMillis - lastCommitTime > 5000) {
consumer.commitAsync(offsetsToCommit);
offsetsToCommit.clear();
lastCommitTime = currentTimeMillis;
}
} catch (Exception e) {
log.error("Failed to commit offsets!", e);
}
}

22. Decoupled multi-threaded model
“Fully decoupled” demo implementation
★ Stopping tasks for revoked partitions
○ Send stop signal to a task, so it can break out its processing loop
○ Wait for task completion using CompletableFuture
○ Commit offsets for revoked partitions (commitSync)
Group rebalance

23. Decoupled multi-threaded model
“FULLY DECOUPLED” DEMO

24. Conclusion
★ Multi-threading is generally hard
★ KafkaConsumer (Java clients) is not thread-safe
★ Key considerations:
○ Offset committing
○ Reacting to group rebalance

25. The end
THANK YOU!
Demo app source code
https://github.com/beegor/ksl20-ib-demo-app
https://github.com/beegor/ksl20-ib-demo-frontend