Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Spring.io Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka
- 1. Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka Marcos Maia & Tim van Baarsen SPRING.IO BARCELONA – MAY 2019
- 2. Introduction Tim van Baarsen Software Engineer at ING @TimvanBaarsen Marcos Maia Software Engineer at ING @thegroo
- 3. ING is active in more than 40 counties 3 https://www.ing.jobs
- 4. Agenda 4 • Use case @ ING • Mifid II • Mobile Investments App • Overall Architecture • Demo • Metrics / Monitoring • Kafka streams • Local development / Code walk through • Lessons learned • Questions
- 5. Disclaimer 5 Both the code and high level architecture we show here today is not the actual product we developed within ING! This example is inspired by our work
- 6. Domain Lingo 6 Tick Price update on stock=
- 7. Mifid II: “Inform customers when a stock drops -10% in price at the end of the day!” 7 Mifid II Regulations ! At the end of the day? "No, in real time!Markets in Financial Instruments Directive II = Regulatory reform financial markets European Union came into effect on January 3, 2018
- 8. Screens of our mobile Investments app 8 price going up! specific price -10% drop in price Recurring alerts
- 9. 2.000 – 12.000 stock updates a second! 9 ! "
- 11. Overall architecture 11 Alert service A P I Matching Service Stocks update receiver Internet “SMS Platform” A P I Receive stock price updates ~ 12.000/sec Publish stock price updates to Kafka Create alerts Delete alerts List alerts Customer receives investment alert Consume updates & match against alerts created by customers In case of a match send a sms to customer Customer alert(s) stored in database Notification Service Schema Registry Zookeeper “Push Platform” A P I !
- 12. Quick Demo 12
- 13. Monitoring - Metrics 13 • Micrometer • Think SLF4J, but for metrics (Jon Schneider) • Application metrics • Out of the box metrics • Business metrics • Vendor Neutral
- 15. Kafka Streams Join 15 Matching Service raw-stock-quote stock-quote-tick stock-quote-meta log-compaction Raw-tick stock-quote-ticker RocksDB Key value ING Raw-tick GOOGL Raw-tick AAPL Raw-tick Key value ING Meta-data GOOGL Meta-data AAPL Meta-data GlobalKTableKStream
- 16. Local development 16 • Docker • Kafka • Zookeeper • Kafka manager • All our Spring Boot applications • Producers for development • Custom Kafka command line producer • Lab project
- 17. Spring Kafka 17 • Producing: • ProducerFactory • KafkaTemplate • “The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances.” • Consuming • Kafka consumers are not thread safe • @KafkaListener • MessageListener (many different interfaces) • Access to ConsumerRecord • Access to manual control to commit the offset • Scale up multiple consumer (threads) within the application
- 18. Spring Kafka 18 • Integration tests • Junit rule: EmbeddedKafkaRule • @EmbeddedKafka • Metrics • Kafka consumer metrics out of the box (Micrometer)
- 19. Lessons learned 19 • Use Spring Kafka after you understand the Kafka APIs • Error handling • (De)serialization -> Apache Avro • Think about # of partitions per topic upfront if possible (routing) • Can your consumers keep up? Can you scale up? • Monitor your topics / streams in production • Based on metrics -> Micrometer
- 20. Lessons learned 20 • Kafka clients are backward compatible with Kafka brokers • Kafka Streams • KTables • GlobalKTables • Interactive Queries • RocksDB build for ssd • Describe • processing.guarantee=exactly_once • Kafka topology • Plan ahead • small & multiple topologies
- 21. Questions 21 !
- 22. Contact Tim van Baarsen Software Engineer @ ING @TimvanBaarsen Marcos Maia Software Engineer @ ING @thegroo