13. 13
Configuration Ground Rules
• Make sure you understand the behavior and what you’re trying to
achieve before changing configs
• Focus on what’s essential for your needs
15. 15
Broker Configs
• JMX metrics
• UncleanLeaderElection
• Retention (Topic and Broker)
• Min InSync Replicas (discussed in Producer section)
16. 16
JMX Metrics
• JMX metrics not exposed remotely by default
• No insight into broker activity without them
17. 17
JMX Metrics
• Set the environment variable JMX_PORT to enable
remote JMX metrics
18. 18
JMX Metrics
• With environment variable JMX_PORT set Kafka
uses these default properties for JMX
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
19. 19
JMX Metrics
• To override/customize the JMX settings, i.e.
provide authentication set the KAFKA_JMX_OPTS
env variable
23. 23
JMX Metrics RequestMetrics
• Requests per second
• Produce - requests from producer to send records
• FetchConsumer – requests from consumer to retrieve
records
• FetchFollower – requests from follower broker to keep in
sync with leader for given topic-partition
28. UncleanLeaderElection
Wait to bring back old leader broker
• Not highly available
• Downtime until old leader recovered
• No potential for missing records
36. 36
Max Poll Interval
max.poll.interval.ms
• max time between poll calls
• If exceeded then consumer kicked out of group, rebalancing
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
//do something with recrods
}
}
37. 37
Max Poll Interval
max.poll.interval.ms = 30000/30 seconds
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
//This loop must complete in under 30 seconds
}
}
38. 38
Max Poll Interval
Processing time for fetched records on consumer takes 45 seconds
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
//Processing in this loop takes 45 seconds this
consumer kicked out of group, and a rebalance is triggered
}
}
39. 39
Max Poll Interval
max.poll.interval.ms solutions -
• Increase the poll interval time – maybe
• Possibly decrease max.poll.records
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
//This loop must complete in under 30 seconds
}
}
40. 40
Max Poll Interval
max.poll.interval.ms solutions -
• Decrease processing time
• Take processing out of loop
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
//This loop must complete in under 30 seconds
}
}
55. 55
Acks All
Producer
Produce waits for leader
broker to acknowledge after
all followers have
acknowledged
Broker (Leader)
Broker
(Follower)
Broker
(Follower)
Got it!
56. 56
Acks All
Producer
Produce waits for leader
broker to acknowledge after
all followers have
acknowledged
Broker (Leader)
Broker
(Follower)
Broker
(Follower)
Got it!
Got it!
66. 66
Retries
Topic A Partition 0
Producer
Retriable
Exception
Use delivery.timeout.ms for limit of time
from sending to receiving
acknowledgement see KIP-91 for details
67. 67
Keeping Records in Order
Topic A Partition 0
Producer
Retriable
Exception
Batch 1
68. 68
Keeping Records in Order
Topic A Partition 0
Producer
Retriable
Exception
Batch 1
Batch 2
69. 69
Keeping Records in Order
Topic A Partition 0
Producer
Retriable
Exception
Batch 1
Batch 2
Batch 1
70. 70
Keeping Records in Order
Topic A Partition 0
Producer
Retriable
Exception
Batch 1
Batch 2
Batch 1
Batch 2 processed before Batch 1 !
71. 71
Keeping Records in Order
Topic A Partition 0
Producer
Retriable
Exception
Batch 1
Batch 2
Batch 1
max.in.flight.requests.per.connection = 1
72. 72
Keeping Records in Order
Topic A Partition 0
Producer
Retriable
Exception
Batch 1
Batch 1
max.in.flight.requests.per.connection = 1
Batch 2
73. 73
Summary
• Configuration is important to the health of a Kafka
Cluster/
• Need to know which knobs to turn
• Measure the before and after
• Try it in development first
77. 77
Thanks!
Stay in Touch!
• https://slackpass.io/confluentcommunity
• https://www.confluent.io/blog/
• Twitter @bbejeck
• We are hiring! https://www.confluent.io/careers/
Editor's Notes
This is me
I’ve worked at Confluent for 1.5 years on Streams team
I authored the book Kafka Streams in Action
Now let’s get started!
First let’s go over what we are going to cover today
This is me
I’ve worked at Confluent for 1.5 years on Streams team
I authored the book Kafka Streams in Action
Now let’s get started!
First let’s go over what we are going to cover today
This is me
I’ve worked at Confluent for 1.5 years on Streams team
I authored the book Kafka Streams in Action
Now let’s get started!
First let’s go over what we are going to cover today
This is me
I’ve worked at Confluent for 1.5 years on Streams team
I authored the book Kafka Streams in Action
Now let’s get started!
First let’s go over what we are going to cover today
This is me
I’ve worked at Confluent for 1.5 years on Streams team
I authored the book Kafka Streams in Action
Now let’s get started!
First let’s go over what we are going to cover today
This is me
I’ve worked at Confluent for 1.5 years on Streams team
I authored the book Kafka Streams in Action
Now let’s get started!
First let’s go over what we are going to cover today
This is me
I’ve worked at Confluent for 1.5 years on Streams team
I authored the book Kafka Streams in Action
Now let’s get started!
First let’s go over what we are going to cover today
This is me
I’ve worked at Confluent for 1.5 years on Streams team
I authored the book Kafka Streams in Action
Now let’s get started!
First let’s go over what we are going to cover today
This is me
I’ve worked at Confluent for 1.5 years on Streams team
I authored the book Kafka Streams in Action
Now let’s get started!
First let’s go over what we are going to cover today
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Image you have a Kafka topic and you need to do a group-by count on it, without Kafka Streams you'd need to do some manual processing to acheive this.
Here is the main method and setting up the consumer and producer and subscribing to two topics
This is more boiler plate work that needs to be done
Image you have a Kafka topic and you need to do a group-by count on it, without Kafka Streams you'd need to do some manual processing to acheive this.
Here is the main method and setting up the consumer and producer and subscribing to two topics
This is more boiler plate work that needs to be done
Image you have a Kafka topic and you need to do a group-by count on it, without Kafka Streams you'd need to do some manual processing to acheive this.
Here is the main method and setting up the consumer and producer and subscribing to two topics
This is more boiler plate work that needs to be done
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Don’t want to blindly increase the max.poll.interval, this setting is piggy backed to other settings on the broker
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Toplogy is a collection of procesing nodes in a graph
A sub-topology is a collection of processing nodes connected by common input topic
Relationship between tasks threads and state stores
Next let's take a look a life before Kafka Streams so we can get a sense of what Kafka Streams is.
Thanks for your time
Stay in touch and use these resources to participate in the community
We have a book signing for Kafka Streams in Action at the Confluent Booth at 4:45 PM today stop
By and pick up a signed copy and check out what’s going on at Confluent
Thanks for your time
Stay in touch and use these resources to participate in the community
We have a book signing for Kafka Streams in Action at the Confluent Booth at 4:45 PM today stop
By and pick up a signed copy and check out what’s going on at Confluent
Thanks for your time
Stay in touch and use these resources to participate in the community
We have a book signing for Kafka Streams in Action at the Confluent Booth at 4:45 PM today stop
By and pick up a signed copy and check out what’s going on at Confluent