Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to 10 Lessons Learned from using Kafka with 1000 microservices - java global summit
Similar to 10 Lessons Learned from using Kafka with 1000 microservices - java global summit (20)
More from Natan Silnitsky
More from Natan Silnitsky (18)
Recently uploaded
Recently uploaded (20)
10 Lessons Learned from using Kafka with 1000 microservices - java global summit
- 1. natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 10 Lessons Learned from using Kafka with 1000 microservices Natan Silnitsky Backend Infra Developer, Wix.com JAVA GLOBAL SUMMIT
- 2. But first… a few Kafka terms
- 3. Kafka Producer Kafka Broker A few Kafka terms @NSilnitsky
- 4. Kafka Producer Topic Partition Partition Partition Kafka Broker Topic Topic Partition Partition Partition Partition Partition Partition A few Kafka terms @NSilnitsky
- 5. Topic TopicTopic Partition Partition Partition Partition Partition Partition Partition Partition Partition Kafka Producer Partition 0 1 2 3 4 5 append-only log A few Kafka terms @NSilnitsky
- 6. Topic TopicTopic Partition Partition Partition Partition Partition Partition Partition Partition Partition Partition 0 1 2 3 4 5 Kafka Consumers 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 A few Kafka terms @NSilnitsky
- 7. Now, Here’s how we use it at Wix.
- 8. >180M registered users (website builders) from 190 countries 5% of all Internet websites run on Wix 3000+ people work at Wix >500B HTTP Requests / Day 6PB of static content At Wix @NSilnitsky
- 9. Kafka Messages per day Microservices (backend) Developers 300M 1075M 500 1,400 100 300* 2017 2020 At Wix
- 10. What do you do when the traffic, meta-data, and amount of developers and use cases grow?
- 11. What do you do when the traffic, meta-data, and amount of developers and use cases grow?
- 12. #1 Common Infra with common features.What do you do when the traffic, meta-data, and amount of developers and use cases grow?
- 13. Kafka Consumer Kafka Producer Kafka Broker Greyhound Wraps Kafka Service A Service B @NSilnitsky
- 14. Kafka Consumer Kafka Producer Kafka Broker Service A Service B Abstract so that it is easy to change for everyone Simplify APIs, with additional features Greyhound Wraps Kafka
- 15. Kafka Consumer Kafka Producer Kafka Broker Service A Service B Scala ZIO + Java API Greyhound Wraps Kafka @NSilnitsky
- 16. Greyhound Wraps Kafka Simple Consumer API - Boilerplate @NSilnitsky
- 17. static void runConsumer() throws InterruptedException { final Consumer<Long, SomeMessage> consumer = createConsumer(); while (true) { final ConsumerRecords<Long, SomeMessage> consumerRecords = consumer.poll(1000); consumerRecords.forEach(record -> { System.out.printf("Record value:%dn", record.value().messageValue); }); consumer.commitAsync(); } } Kafka Consumer API
- 18. static void runConsumer() throws InterruptedException { final Consumer<Long, SomeMessage> consumer = createConsumer(); while (true) { final ConsumerRecords<Long, SomeMessage> consumerRecords = consumer.poll(1000); consumerRecords.forEach(record -> { System.out.printf("Record value:%dn", record.value().messageValue); }); consumer.commitAsync(); } } Kafka Consumer API
- 19. static void runConsumer() throws InterruptedException { final Consumer<Long, SomeMessage> consumer = createConsumer(); while (true) { final ConsumerRecords<Long, SomeMessage> consumerRecords = consumer.poll(1000); consumerRecords.forEach(record -> { System.out.printf("Record value:%dn", record.value().messageValue); }); consumer.commitAsync(); } } Kafka Consumer API
- 20. static void runConsumer() throws InterruptedException { String topic = "some-topic"; MessageHandler<SomeMessage> handler = message -> System.out.printf("Record value:%dn", message.messageValue); GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler); } Greyhound Consumer API * No explicit commit
- 22. static void runConsumer() throws InterruptedException { final Consumer<Long, SomeMessage> consumer = createConsumer(); while (true) { final ConsumerRecords<Long, SomeMessage> consumerRecords = consumer.poll(1000); consumerRecords.forEach(record -> { System.out.printf("Record value:%dn", record.value().messageValue); }); consumer.commitAsync(); } } Kafka Consumer API
- 23. static void runConsumer() { MessageHandler<PurgeSite> handler = ... GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler) .withGroup("some-group") .withMaxParallelism(40); } Greyhound Consumer @NSilnitsky
- 24. Kafka Broker Site Chat-bot-m essages Topic Greyhound Consumer Kafka Consumer SCALA ZIO FIBERS + QUEUES 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 (THREAD-SAFE) PARALLEL CONSUMPTION @NSilnitsky
- 25. + Retries! ...what about Error handling? Greyhound Wraps Kafka Simple Consumer API Thread-safe Parallel Consumption
- 26. static void setupConsumer() { ConsumeRetryPolicy retryPolicy = ConsumeRetryPolicy.aRetryPolicy( Retries.fromBackoffs(Duration.ofSeconds(1), Duration.ofMinutes(10), ...)); GreyhoundConsumer renewConsumer = GreyhoundConsumer.aGreyhoundConsumerSpec( topic, renewHandler) .withGroup("some-group") .withRetry(retryPolicy); } Greyhound Consumer @NSilnitsky
- 27. Kafka Broker renew-sub-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Greyhound Consumer Kafka Consumer FAILS TO READ @NSilnitsky
- 28. Kafka Broker renew-sub-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-0 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-1 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Inspired by Uber RETRY! Greyhound Consumer Kafka Consumer RETRY PRODUCER @NSilnitsky
- 29. #2 Retry Topics will cause your cluster to grow faster. 😐 What do you do when the traffic, meta-data, and amount of developers and use cases grow? #1 Common Infra
- 30. Kafka Broker renew-sub-topic-retry-0 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-1 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Kafka Broker renew-sub-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-N 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 RETRY! Inspired by Uber @NSilnitsky
- 31. Kafka Broker renew-sub-topic-retry-0 0 1 2 3 4 5 renew-sub-topic-retry-1 0 1 2 3 4 5 Kafka Broker renew-sub-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-N RETRY! Inspired by Uber 0 2 3 4 51 @NSilnitsky
- 32. Retries same message on failure * lag BLOCKING POLICY HANDLER Kafka Broker source-control- update-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Greyhound Consumer Kafka Consumer build-log-service
- 34. + Context Propagation Super cool for us Greyhound Wraps Kafka Simple Consumer API Thread-safe Parallel Consumption Scheduled retries & blocking handler
- 35. CONTEXT PROPAGATION Language User Type USER REQUEST METADATA Sign up Site-Members Service Browser Geo ... @NSilnitsky
- 38. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #3 Self-service tooling and documentation. #2 Retry Topics - bigger cluster #1 Common Infra
- 39. Our team
- 41. 1. Kafka CLI scripts (+ flags) 2. Yahoo Kafka Manager (OSS) 3. Confluent Control Centre 4. A control plane of our own Self-service Tools
- 49. Self-service Docs How do I investigate this lag? How do I add retries on errors? 1. Github Readme for Greyhound code 2. Internal StackOverflow Q&A 3. Slack bot that answers without you
- 53. protobuf internal API site Self-service Docs Discover topics and message structure. Message structure defined in... And exposed with... @NSilnitsky
- 54. protobuf internal API site Self-service Docs Discover topics and message structure. Or… avro schema registry Message structure defined in... And exposed with... @NSilnitsky
- 55. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #4 Testing needs to be easy. v #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra
- 56. Greyhound Testkit Kafka Broker ZooKeeper Test Environment Scala Test { } Testkit included! @NSilnitsky
- 57. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #5 Async event driven monitoring is less trivial. #4 Easy Testing #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra
- 59. @NSilnitsky
- 60. #6 Proactive broker maintenance. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #5 Non-trivial Monitoring #4 Easy Testing #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra
- 61. ● Add brokers when needed ● Split clusters when needed ● Delete unused topics ● Avoid hard failures As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions. https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions Don’t let brokers break... @NSilnitsky
- 62. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #7 Avoid active-active ordered produce (write) #5 Non-trivial Monitoring #4 Easy Testing #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra #6 Proactive broker maintenance
- 63. Kafka Broker Producer Update my site Browser service 1 1 DC1 @NSilnitsky
- 64. Kafka Broker Update my siteBrowser Kafka Broker Producer service 1 2 21 DC1 DC2 Producer @NSilnitsky
- 65. Kafka Broker Browser Kafka Broker Producer 1 122 2 DC1 DC2 Producer 1 REPLICATION Out of Order @NSilnitsky
- 66. Kafka Broker Update my site Browser Kafka Broker Producer 1 2 Producer DC1 DC2 Best Practice @NSilnitsky
- 67. Kafka Broker Browser 1 2 Producer Kafka Broker 1 2 DC1 DC2 REPLICATION @NSilnitsky
- 68. Kafka Broker Browser 1 2 Producer Kafka Broker 1 2 DC1 DC2 Consumer REPLICATION @NSilnitsky
- 69. Kafka Broker Browser 1 2 Producer Kafka Broker 1 2 DC1 DC2 Consumer REPLICATION @NSilnitsky
- 70. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #8 Avoid using Kafka SDK directly in nodeJs #7 Avoid active-active write #5 Non-trivial Monitoring #4 Easy Testing #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra #6 Proactive broker maintenance
- 71. Greyhound Kafka Broker Producer Consumer Scala services NodeJS services ConsumerProducer Single Event loop @NSilnitsky
- 72. Greyhound Kafka Broker Producer Consumer Scala services NodeJS services ConsumerProducer Single Event loop ✘ @NSilnitsky
- 73. * memory Greyhound Kafka Broker Consumer Sidecar (Or REST Proxy) gRPC NodeJS services Single Event loop
- 74. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #9 Consume and project #8 Avoid nodeJs SDK #7 Avoid active-active write #5 Non-trivial Monitoring #4 Easy Testing #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra #6 Proactive broker maintenance
- 76. Read + Writes Large MetaSite Object Request overload (~1M RPM requests) MetaSite Site installed Apps? RPC Wix Bookings Wix Restaurants Site version? Site owner? RPC RPC Wix Stores @NSilnitsky
- 77. Read + Writes Large MetaSite Object Request overload (~1M RPM requests) MetaSite Site installed Apps? RPC Wix Bookings Wix Restaurants Site version? Site owner? RPC RPC Wix Stores @NSilnitsky
- 79. Kafka Broker Filter Site installed Apps Updated! Installed Apps Context 1. Produce to Kafka 2. Consume and Project Producer MetaSite Site Updated! Consumer Reverse lookup writer
- 80. Kafka Broker Filter Site installed Apps Updated! Installed Apps Context 1. Produce to Kafka 2. Consume and Project Producer MetaSite Site Updated! Consumer Reverse lookup writer Reverse lookup reader RPC RPC RPC 3. Split Read from Write
- 81. Kafka messaging is event driven. It is only relevant to service-service communications, not for browser-server interactions, where a user is waiting, right? @NSilnitsky
- 82. Kafka messaging is event driven. It is only relevant to service-service communications, not for browser-server interactions, where a user is waiting, right? Wrong. @NSilnitsky
- 83. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #10 WebSockets are Kafka’s best friend #9 Consume and project #8 Avoid nodeJs SDK #7 Avoid active-active write #5 Non-trivial Monitoring #4 Easy Testing #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra #6 Proactive broker maintenance
- 84. Completely distributed and event driven Kafka WebSockets + = @NSilnitsky
- 85. Kafka Broker Consumer Browser Producer Contacts Importer Contacts Jobs Web Sockets Service Use Case: Long-running async business process @NSilnitsky
- 86. Kafka Broker Subscribe for notifications ConsumerProducer Use Case: Long-running async business process Contacts Importer Contacts Jobs Browser Web Sockets Service @NSilnitsky
- 87. Kafka Broker Import CSVs ConsumerProducer Use Case: Long-running async business process Contacts Importer Contacts Jobs Browser Web Sockets Service @NSilnitsky
- 88. Web Sockets Service Kafka Broker Consumer * distributed Use Case: Long-running async business process Producer Contacts Jobs Browser done! Contacts Importer
- 89. So, What do you do when the traffic, meta-data, and amount of developers and use cases grow?
- 90. Wix Created an entire ecosystem to support large-scale Kafka-related needs.
- 91. A Java/Scala high-level SDK for Apache Kafka. 0.1 is out! github.com/wix/greyhound
- 92. Thank You JAVA GLOBAL SUMMIT natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
- 94. Q&A JAVA GLOBAL SUMMIT natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil