Bernhard Hopfenmüller, Senior IT Consultant @ ATIX AG

1. v = 346,39 m/s

2. v = 346,39 m/s Speed of sound at 25 °C
v = 349,29 m/s Speed of sound at 30 °C
front to audience: 2 m
latency: 5.7 ms
latency (back of room): 15 ms
#atix #streamingworkshop

3. v = 346,39 m/s Speed of sound at 25 °C
v = 349,29 m/s Speed of sound at 30 °C
front to audience: 2 m
latency: 5.7 ms
latency (back of room): 15 ms
#atix #streamingworkshop

4. v = 346,39 m/s Speed of sound at 25 °C
v = 349,29 m/s Speed of sound at 30 °C
front to audience: 2 m
latency: 5.7 ms
latency (back of room): 15 ms
#atix #streamingworkshop

5. v = 346,39 m/s Speed of sound at 25 °C
v = 349,29 m/s Speed of sound at 30 °C
front to audience: 2 m
latency: 5.7 ms
latency (back of room): 15 ms
#atix #streamingworkshop

6. v = 346,39 m/s Speed of sound at 25 °C
v = 349,29 m/s Speed of sound at 30 °C
front to audience: 2 m
latency: 5.7 ms
latency (back of room): 15 ms
#atix #streamingworkshop

7. take home message: I am faster than my ISP!
Source: speedcheck.org
#atix #streamingworkshop

8. How to tune Kafka for
production
Dr. Bernhard Hopfenmüller
24. Juli 2019
#atix #streamingworkshop

9. What do you need?
Latency Throughput
Durability Availability
???
IoT
#atix #streamingworkshop

10. whoami
Bernhard Hopfenmüller
Senior IT Consultant @ ATIX AG
current latency 8 ms
hopfenmueller@atix.de
github.com/Fobhep
Twitter: @fobhep
IRC: Fobhep
#atix #streamingworkshop

11. Throughput
#atix #streamingworkshop

12. Throughput: Batching
increase batch
size
increase linger
time
#atix #streamingworkshop

13. Throughput: Compression
Compress FULL
batches
lz4, snappy, zstd, gzip
be consistent - avoid
recompression!!
#atix #streamingworkshop

14. Throughput: Replication Guarantees
did my message
arrive?
acks=1 (default)
#atix #streamingworkshop

15. Throughput: Memory
Increase producer
buffer memory
Increase consumer
fetch size
#atix #streamingworkshop

16. Throughput: Consumer Groups
Parallelize data
procession
#atix #streamingworkshop

17. Throughput: JVM
-Xms6g -Xmx6g -XX:MetaspaceSize=96m -XX:+UseG1GC
-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35
-XX:G1HeapRegionSize=16M
-XX:MinMetaspaceFreeRatio=50
-XX:MaxMetaspaceFreeRatio=80
Too long JVM GC pause time
could cause
Zookeeper session timeout
impact throughput
#atix #streamingworkshop

18. Throughput: Partitions
More Partition == Higher
Troughput?!
yes, but!
More Open File Handlers
#atix #streamingworkshop

19. Throughput: Partitions
More Partition == Higher
Troughput?!
yes, but!
More Open File Handlers
More Memory needed for
Producer and Consumer
#atix #streamingworkshop

20. Throughput: Summary
Producer:
batch.size: 100000 – 200000 (default 16384)
linger.ms: 10 – 100 (default 0)
compression.type: lz4 (default none, i.e., no compression)
acks: 1 (default 1)
buffer.memory: increase according to partitions (default 33554432)
Consumer:
fetch.min.bytes: ~100000 (default 1)
#atix #streamingworkshop

21. Latency
#atix #streamingworkshop

22. Latency: Batching
small batches!
linger.ms=0
not! no
batching
#atix #streamingworkshop

23. Latency: Compression
no compression
reduces bandwidth
utilization
less cpu load
but! benchmark,
codecs might be
good after all
#atix #streamingworkshop

24. Latency: Replication Guarantees
did my message
arrive?
#atix #streamingworkshop

25. Latency: Replication Guarantees
acks=1 (0 if you are
brave)
#atix #streamingworkshop

26. Latency: Memory
fetch.min.bytes=1
(fetch data asap)
#atix #streamingworkshop

27. Latency: Kafka Streams - External Queries
Source: Confluent
Query external
databases
high latency
#atix #streamingworkshop

28. Latency: Kafka Streams - Table Stream Duality
Source: Confluent
A table is a stream is
a table ...
#atix #streamingworkshop

29. Latency: Kafka Streams - Local Table Joins
Source: Confluent
Include external data
via Connect
join Tables locally
with streams
decrease latency
#atix #streamingworkshop

30. Latency: Kafka Streams - Topologies
Source: Confluent
Input from Kafka
Operations (filter map
aggregation join)
Back to Kafka
#atix #streamingworkshop

31. Latency: Kafka Streams - Topologies
Source: Confluent
Repartitioning may
occur
increase latency
#atix #streamingworkshop

32. Latency: Kafka Streams - Topologies
config.setProperty(
StreamsConfig.TOPOLOGY_OPTIMIZATION,
StreamsConfig.OPTIMIZE);
Avoid unnecessary
repartitioning
decrease latency
#atix #streamingworkshop

33. Latency: Partitions
Source: Confluent
More Partition == Higher
Troughput?!
yes, but!
broker has single thread for
replication (default)
latency might increase
limit topics per broker
increase num.replica.fetchers
#atix #streamingworkshop

34. Latency: Summary
Producer:
linger.ms: 0 (default)
compression.type: none (default)
acks:1 (default)
Consumer:
fetch.min.bytes=1
Streams:
StreamsConfig.TOPOLOGY_OPTIMIZATION StreamsConfig.OPTIMIZE
use kafka connect and local joins
Broker:
num.replica.fetchers: increase (default 1)
#atix #streamingworkshop

35. Durability
#atix #streamingworkshop

36. Durability: Producer Replication
Source: Confluent
choose a higher replication
factor
don’t forget internal topics!
__consumer_offsets
internal streams topics
exactly once semantics
#atix #streamingworkshop

37. Durability: Producer Acks
acks=all
#atix #streamingworkshop

38. Durability: Producer Retries
Source: Confluent
retries
duplicates
ordering
problems
#atix #streamingworkshop

39. Durability: Producer Idempotency
Source: Confluent
produce
idempotent!
#atix #streamingworkshop

40. Durability: Delivery Semantics
at-least-once delivery
at-most-once delivery
exactly-once delivery (Kafka only!)
#atix #streamingworkshop

41. Durability: Exactly Once Semantic
(
Send Data with PID and increasing seq
Broker: Check if seq > seq_old
write only if true
)
1. Idempotent
Producer
#atix #streamingworkshop

42. Durability: Exactly Once Semantic
(
Send Data
Read Data (Write to Commit Log)
)
Either write all of that or none
Source: Confluent
1. Idempotent
Producer
2. Multiple Partition
Transactions
#atix #streamingworkshop

43. Durability: Exactly Once Semantic
producer.initTransactions();
[...]
while (true) {
ConsumerRecords records = consumer.poll(Long.MAX_VALUE);
producer.beginTransaction();
for (ConsumerRecord record : records)
producer.send(producerRecord(“outputTopic”, record));
producer.sendOffsetsToTransaction(currentOffsets(consumer), group);
producer.commitTransaction();
}
Source: Confluent
1. Idempotent
Producer
2. Multiple
Partition
Transactions
#atix #streamingworkshop

44. Durability: Streams EOS
StreamsConfig.PROCESSING_GUARANTEE_CONFIG:
StreamsConfig.EXACTLY_ONCE
StreamsConfig.REPLICATION_FACTOR_CONFIG: 3 (default 1)
EOS for Streams very
simple
don’t forget
replication of streams
topics
#atix #streamingworkshop

45. Durability: Consumer
no auto commit!
enable.auto.commit false
for EOS (isolation.level =
read_commited)
#atix #streamingworkshop

46. Durability: Replica
min.insync.replica > 1
#atix #streamingworkshop

47. Durability: Summary
Producer:
replication.factor=3 (topic override available)
acks=all (default 1)
enable.idempotence=true (default false), to handle message duplication and ordering
Consumer:
enable.auto.commit=false (default true)
isolation.level=read_committed (for EOS transactions)
#atix #streamingworkshop

48. Durability: Summary
Streams:
StreamsConfig.REPLICATION_FACTOR_CONFIG: 3 (default 1)
StreamsConfig.PROCESSING_GUARANTEE_CONFIG: StreamsConfig.EXACTLY_ONCE
Broker:
default.replication.factor=3 (default 1)
auto.create.topics.enable=false (default true)
min.insync.replicas=2 (default 1); topic override available
broker.rack: rack of the broker (default null)
#atix #streamingworkshop

49. Availability
Source: Confluent
#atix #streamingworkshop

50. Availability: Replica
min.insync.replicas = 1
unclean.leader.election.enable =
true
#atix #streamingworkshop

51. Availability: Log Recovery
broker start: scan all log files
to sync
usually one thread per data
dir
increase to number of dirs in
log.files
#atix #streamingworkshop

52. Availability: Stream State Restoration
Source: Confluent
store data for stateful stream
ops
create copies of state for
reusing
num.standby.replicas > 0
#atix #streamingworkshop

53. Availability: Summary
Consumer:
session.timeout.ms: as low as feasible (default 10000)
Streams:
StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG: 1 or more(default 0)
Broker:
unclean.leader.election.enable=true (default false); topic override available
min.insync.replicas=1 (default 1); topic override available
num.recovery.threads.per.data.dir: number of directories in log.dirs (default 1)
#atix #streamingworkshop

54. And now?: Next steps
1. Decide what you want
Repeat:
2. Benchmark - Control Center!
3. Act on results
4. MONITOR! - Control Center!
#atix #streamingworkshop

55. And now?: Sources and Material
What is Kafka https://www.heise.de/select/i-
x/2019/4/1553935521978043
Kafka Optimization https://www.confluent.io/white-
paper/optimizing-your-apache-kafka-deployment/
Kafka Partition number https://www.confluent.io/blog/how-choose-
number-topics-partitions-kafka-cluster
RTFM https://docs.confluent.io/current/administer.html
#atix #streamingworkshop

56. And now?: Talk to us
Datacenter
Automation
Deploy
ConfigureRelease
#atix #streamingworkshop

57. Let’s optimize our latency!
#atix #streamingworkshop