Streaming all over the World: Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka.
Learn about various case studies for event streaming with Apache Kafka across industries. The talk explores architectures for real-world deployments from Audi, BMW, Disney, Generali, Paypal, Tesla, Unity, Walmart, William Hill, and more. Use cases include fraud detection, mainframe offloading, predictive maintenance, cybersecurity, edge computing, track&trace, live betting, and much more.
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
1. Streaming all over the World
Real-Life Use Cases & Architectures for Event Streaming
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
2. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming
3. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Global Scale
Real-time
Persistent Storage
Stream Processing
Apache Kafka
The De-facto Standard for Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache
Kafka
4. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate
to Cloud
Mitigate
Risk (protect
money)
Key Drivers
Strategic
Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation &
improved in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous
System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital
One
Developer Velocity - Building
Stateful Financial Applications with
Kafka Streams: Funding Circle
Detect Fraud & Prevent Fraud in
Real Time: PayPal
Kafka as a Service - A Tale of
Security and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)
5. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences
6. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences
7. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
8. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
AI/ML
Modernized security information and event management (SIEM)
Filter, transform
aggregate
APP SIEM Index
Search
Curated streams
Forensic
Archive
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
Payment Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
9. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/transforming-security-posture-with-innovations-in-data-intelligence-paper.pdf
10. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
“… rescue data off of the mainframe, in a cloud native,
microservice-based fashion … [to] … significantly reduce the
reads on the mainframe, saving RBC fixed infrastructure
costs (OPEX). RBC stayed compliant with bank regulations
and business logic, and is now able to create new applications
using the same event-based architecture.”
Mainframe Offloading
for massive cost-savings
https://www.confluent.io/customers/rbc/
11. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Mainframe Offloading
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
MIPS / MSU
MIPS / MSU
MIPS / MSU
Read
No MIPS / MSU
12. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Mainframe Replacement
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
MIPS / MSU
MIPS / MSU
MIPS / MSU
Read
No MIPS / MSU
13. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences
14. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Integration Platform
for legacy and modern technologies
https://www.jug.ch/events/slides/190918_Microservices_and_Kafka_on_OpenShift.pdf
15. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent https://www.jug.ch/events/slides/190918_Microservices_and_Kafka_on_OpenShift.pdf
Integration Platform
for legacy and modern technologies
16. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences
17. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Food Value Chain
IoT-Based and Data-Driven
Single source of truth
across the food value chain
(in the factories, and across regions)
Business critical
operations
(tracking, calculations, alerts, …)
https://www.confluent.io/blog/creating-iot-based-data-driven-food-value-chain-with-confluent-cloud/
18. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and ConfluentReal-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cross-Company Supply Chain Integration
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Cluster Linking
Tier 2
Supplier
OEM Streaming integration
between companies
API Management
(REST et al) is not
appropriate for
streaming data
Infosec and politics are
your biggest hurdle
Tier 1
Supplier
19. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences
20. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Tesla
Trillions of messages per day for IoT use cases
https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/
https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/
21. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
BMW Group
Industry-ready NLP Service Framework Based on Kafka
https://www.confluent.io/kafka-summit-lon19/industry-ready-nlp-service-framework-kafka/
22. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model BModel A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with Kafka, Tiered Storage and TensorFlow IO
https://github.com/tensorflow/io
22
Model X
(at a later time)
23. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences
24. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Four
Telecom
scenarios
for 2030
https://www2.deloitte.com/content/dam/Deloitte/pl/
Documents/Reports/pl_Deloitte_TMT_Telco_2030.pdf
25. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
From Monoliths to decoupled, independent services
https://www2.deloitte.com/content/dam/Deloitte/it/Documents/technology-
media-telecommunications/Next%20Gen%20Telco%20Architecture_2017_final.pdf
(Network Functions Virtualization)
26. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Hotstar
OTT for millions of cricket fans in India
https://www.confluent.io/kafka-summit-san-francisco-2019/scaling-for-indias-cricket-hungry-population/
27. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
OSM - Open Source MANO
Interoperability among different
service provider NFV infrastructures and OSS systems
https://osm.etsi.org/news-events/blog/30-open-source-mano-addressing-interoperability-challenge-in-nfv
28. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent https://osm.etsi.org/
29. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences
30. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Walmart
Scalable and Reliable Event Streaming
at World’s Largest Retailer
https://kafka-summit.org/sessions/kafka-meets-scaling-reliability-needs-worlds-largest-retailer-walmart-story/
31. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Ride-Sharing
More than just Messaging! Data Correlation in Real-Time
for map-matching, ETA, cost calculation, and much more…
https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006
32. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Track, manage, and locate
tools and other equipment
anytime and anywhere from
the warehouse to the jobsite
https://www.confluent.io/customers/bosch/
https://events.confluent.io/online-talks/bosch-power-toolse-nables-real-time-analytics-on-iot-event-streams
33. DB Musterfirma | Vorname Name | Abteilung | Datum ("Einfügen > Kopf- und Fußzeile")
33Deutsche Bahn AG | Reisendeninformation
Consistent
real-time information
for travellers
across Germany
RI-Plattform
34. DB Musterfirma | Vorname Name | Abteilung | Datum ("Einfügen > Kopf- und Fußzeile")
34
Customer timetable
Operational
timetable
Assignments
Railway station
knowledge
Dispositions
Train positions
Matching
Aggregation
Consolidation
Apache
Kafka
Analysis
Railway station
Trains
Mobile Apps
Employees
Deutsche Bahn AG | Reisendeninformation
RI-Plattform
35. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences
36. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Big Fish Games
Real-time analytics of game telemetry data for live operations
(aka increasing revenue while the player plays the game)
https://www.confluent.io/kafka-summit-sf18/how-big-fish-games-developed-real-time-analytics/
Casual and mid-core games.
2.5 billion games to customers in
150 countries, representing over
450 unique mobile games and
over 3,500 unique PC games.
https://www.confluent.io/kafka-summit-sf18/how-big-fish-games-developed-real-time-analytics/
37. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Unity Ads - Monetization Network
• In 2019, content installed 33 billion times
reaching 3 billion devices worldwide
• Real-time 3D development platform
• Unity is a data-driven company
• Single common data pipeline for analytics, R&D,
monetization, cloud services, etc. for real-time
and batch processing
• One of the largest monetization networks
in the world
• Reward players for watching ads
• Incorporate banner ads
• Incorporate Augmented Reality (AR) ads
• Playable ads
• Cross-Promotions
https://www.confluent.io/blog/how-unity-uses-confluent-for-real-time-event-streaming-at-scale/
38. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
William Hill
From a Monolith to a flexible, scalable microservice architecture
• Kafka as central, reliable
streaming infrastructure
• Kafka for messaging,
storage, cache and
processing of data
• Independent decoupled
microservices
• Decoupling and replayability
• Technology independence
• High throughput + low
latency + real time
https://www.codemesh.io/codemesh2015/peter-morgan
https://www.confluent.io/kafka-summit-london18/building-low-latency-high-throughput-pipelines-with-kafka-from-scratch/
39. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
William Hill
The trading platform for millions of bets every day
• Kafka is the heart of all
events and transactions
• “process-to-process”
execution in real time
• Integration with analytic
models for real time
machine learning
• Various data sources and
data sinks (real time,
batch, request-response)
https://www.codemesh.io/codemesh2015/peter-morgan
https://www.confluent.io/kafka-summit-london18/building-low-latency-
high-throughput-pipelines-with-kafka-from-scratch/
40. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Betting delay and approval in live bets
Synthetic delay to calculate risk, adjust odds, increase margin, reject bet if “too good”
Stateful Correlation of Events
Live Bet
(HTTP from Mobile App)
Time
Betting Engine
(ksqlDB)
41. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences
42. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Pharma, Life Sciences and Healthcare
Event Streaming to improve traditional and to build new use cases
Streams Processing / AI / ML
Clinical Trials
Patents,
Text etc
Structured &
unstructured
Data
IoT & Business
Applications
Multi-Hybrid-
Cloud
43. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Bayer AG
Connectivity
+
On Premise and
Cloud
+
Hybrid Real Time
Replication at
Scale
Cloud first strategy and started a multi-year transition
to the cloud with a Kafka-based cross-datacenter data hub
https://www.confluent.io/kafka-summit-sf18/bringing-streaming-data-to-the-masses
44. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Recursion Pharmaceutical
Real-time
Event Streaming
+
Machine Learning
Recursion Pharmaceutical
Accelerate drug discovery
https://www.confluent.io/customers/recursion
https://www.confluent.io/kafka-summit-san-francisco-2019/discovering-drugs-with-kafka-streams
45. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Drug Discovery
in manual and slow, bursty batch mode, not scalable
https://www.confluent.io/customers/recursion
https://www.confluent.io/kafka-summit-san-francisco-2019/discovering-drugs-with-kafka-streams
46. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Drug Discovery
in automated, scalable, reliable real-time mode
https://www.confluent.io/customers/recursion
https://www.confluent.io/kafka-summit-san-francisco-2019/discovering-drugs-with-kafka-streams
47. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Digital Image
Processing
(e.g. noise
reduction)
Streaming Analytics for
Drug Discovery in Real Time at Scale
Real Time
Integration
Layer
Batch
Reporting
Platform
BI
DashboardEvent
Streaming
Platform
Real Time
Integration
Layer
Laboratory
Streaming Platform
Other Components
Automated
Drug Analysis
All
Data
Processed
Images
Ingest
Images
Human
Intelligence
Data Processing
(e.g. filtering)
Stateful Workflow
Orchestration
48. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Confluent Platform
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF
CHOICE
COMMITTER-DRIVEN
EXPERTISE
PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based
Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete
Engagement Model
Revenue / Cost / Risk
Impact
TCO / ROI
Executive Buyer