Kafka is the bedrock of Wix’s distributed Mega Microservices system. Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices. In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These Include: * How to increase dev velocity of event-driven style code. * How to optimize working with Kafka in polyglot setting * How to migrate from request-reply to event-driven * How to tackle multiple DCs environment.

17. val consumer: KafkaConsumer[String, SomeMessage] =
createConsumer()
def pollProcessAndCommit(): Unit = {
val consumerRecords = consumer.poll(1000).asScala
consumerRecords.foreach(record => {
println(s"Record value: ${record.value.messageValue}")
})
consumer.commitAsync()
pollProcessAndCommit()
}
pollProcessAndCommit()
* Broker location, serde

18. val consumer: KafkaConsumer[String, SomeMessage] =
createConsumer()
def pollProcessAndCommit(): Unit = {
val consumerRecords = consumer.poll(1000).asScala
consumerRecords.foreach(record => {
println(s"Record value: ${record.value.messageValue}")
})
consumer.commitAsync()
pollProcessAndCommit()
}
pollProcessAndCommit()

19. val consumer: KafkaConsumer[String, SomeMessage] =
createConsumer()
def pollProcessAndCommit(): Unit = {
val consumerRecords = consumer.poll(1000).asScala
consumerRecords.foreach(record => {
println(s"Record value: ${record.value.messageValue}")
})
consumer.commitAsync()
pollProcessAndCommit()
}
pollProcessAndCommit()

20. val handler: RecordHandler[Console, Nothing,
String, SomeMessage] =
RecordHandler { record =>
zio.console.putStrLn(record.value.messageValue)
}
GreyhoundConsumersBuilder
.withConsumer(GreyhoundConsumer(
topic = "some-group",
group = "group-2",
handle = handler))
* No explicit commit, broker location

22. val consumer: KafkaConsumer[String, SomeMessage] =
createConsumer()
def pollProcessAndCommit(): Unit = {
val consumerRecords = consumer.poll(1000).asScala
consumerRecords.foreach(record => {
println(s"Record value: ${record.value.messageValue}")
})
consumer.commitAsync()
pollProcessAndCommit()
}
pollProcessAndCommit()

25. val retryPolicy = RetryPolicy.default(group,
1.second, 10.minutes)
GreyhoundConsumersBuilder
.withConsumer(GreyhoundConsumer(
topic = "some-group",
group = "group-2",
handle = handler,
retryPolicy = retryPolicy))

28. 😐

34. Language
User
Type
Geo

60. ●
●
●
●
As a rule of thumb, we recommend each broker to have up to 4,000 partitions
and each cluster to have up to 200,000 partitions.

62. 1
1

63. 1 2
21

64. 1 122
21

65. 1 2

66. 1 2 1 2

67. 1 2 1 2

68. 1 2 1 2

71. ✘

72. gRPC