Apache Kafka and Event Streaming are two of the most relevant buzzwords in tech these days. Ever wonder what the predicted TOP 5 Event Streaming Architectures and Use Cases for 2021 are? Check out the following presentation. Learn about edge deployments, hybrid and multi-cloud architectures, service mesh-based microservices, streaming machine learning, and cybersecurity.
On-demand video recording: https://videos.confluent.io/watch/XAjxV3j8hzwCcEKoZVErUJ
Top 5 Event Streaming Use Cases for 2021 with Apache Kafka
1. The Top 5 Event Streaming
Use Cases & Architectures in 2021
Hybrid Architectures, Edge Computing, Machine Learning, Cybersecurity, Service Mesh
Kai Waehner
Field CTO
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
2. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Gartner Top
Strategic Technology
Trends for 2021
https://www.gartner.com/smarterwithgartner/gartner-top-strategic-technology-trends-for-2021/
3. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming
4. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Global Scale
Real-time
Persistent Storage
Stream Processing
Data Integration
Apache Kafka
The De-facto Standard for Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache
Kafka
5. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate
to Cloud
Mitigate
Risk (protect
money)
Key Drivers
Strategic
Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation &
improved in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous
System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital
One
Developer Velocity - Building
Stateful Financial Applications with
Kafka Streams: Funding Circle
Detect Fraud & Prevent Fraud in
Real Time: PayPal
Kafka as a Service - A Tale of
Security and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)
6. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
7. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
8. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Why Kafka in Multiple Data Centers?
* Not a representative survey J
** ‘Many DCs’ does NOT necessarily mean more than one Kafka Cluster
9. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Disaster Recovery – RPO and RTO
RPO = Recovery Point Objective
RTO = Recovery Time Objective
10. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Disaster Recovery @ JPMorgan
https://www.confluent.io/kafka-summit-san-francisco-2019/secure-kafka-at-scale-in-true-multi-tenant-environment
11. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cluster Linking
• Hybrid-cloud and multi-cloud
• No additional infrastructure (such as Kafka Connect or MirrorMaker)
• Just configuration
• Regional or global
12. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
What is the right Hybrid Kafka Architecture for you?
(Hint: This is hard à Let’s guide you by our experts)
12
Latency
> 50ms
Latency
< 50ms
RTO = 0 RTO > 0 RPO = 0 RPO > 0 Single Region Multi-Region Global
Stretched Cluster
x x x x
Replicator
x x x x* x
Cluster-Linking
x x x x* x
MRC Sync
x x x x**
MRC Observer
x x x x**
* With a stretched cluster in a single region, you still have RTO & RPO = 0
** Requires 3 regions minimum
13. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Kafka as a Service – Fully Managed?
Infrastructure
management
(commodity)
Scaling
● Upgrades (latest stable version of Kafka)
● Patching
● Maintenance
● Sizing (retention, latency, throughput, storage, etc.)
● Data balancing for optimal performance
● Performance tuning for real-time and latency requirements
● Fixing Kafka bugs
● Uptime monitoring and proactive remediation of issues
● Recovery support from data corruption
● Scaling the cluster as needed
● Data balancing the cluster as nodes are added
● Support for any Kafka issue with less than X minutes response time
Infra-as-a-Service
Harness full power of Kafka
Kafka-specific
management
Platform-as-a-Service
Evolve as you need
Future-proof
Mission-critical reliability
Most Kafka-as-a-Service offerings are partially-managed
Kafka as a Service should be a serverless experience with consumption-based pricing!
14. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
15. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
What is the “Edge” for Kafka?
• Edge is NOT a data center
• Kafka clients AND the Kafka broker(s)
• Offline business continuity
• Often 100+ locations
• Low-footprint and low-touch
• Hybrid integration
Example:
Single broker, 1 GB Ram
100 MB/sec
16. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Hybrid Architecture
17. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Food
Inventory Loyalty
System
Traveler
Information
Orders Upsell to
first class
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Updated
SchedulesEvent Streaming
at the Edge
18. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Data Processing
at the Edge
Time
P
C1
C2
C3
Know-your-customer
Loyalty app, predictive behavior, …
Estimated
time of arrival
Connect to the
gaming server
for kids
Play games, earn rewards, communicate
with other kids in the train, …
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
19. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Edge Kafka @ Royal Caribbean
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
20. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Edge Integration and Analytics @ WPX Energy
Edge processing and
replication to the cloud
in real-time at scale
in the oil&gas industry
21. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
22. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Microservices to the rescue?
• Significant Operations Overhead
• Substantial DevOps Skills Required
• Implicit Interfaces
• Duplication Of Effort
http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
• Distributed System Complexity
• Asynchronicity Is Difficult
• Testability Challenges
22
23. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Microservices can lead to Death-star Architectures
Netflix: https://www.slideshare.net/brucewong3/the-case-for-chaos
Twitter: https://twitter.com/adrianco/status/441883572618948608
Hail-o: http://www.sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-3/
450+ microservices 500+ microservices 500+ microservices
23
24. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
24
25. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Kafka Connect
Kafka Cluster
CRM
Integration
Domain-Driven Design (DDD) for your Microservice Architecture
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
25
26. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
26
27. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cloud-Native Deployment leveraging Kubernetes
27
28. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Service Mesh
A microservice pattern to move visibility, reliability, and
security primitives for service-to-service communication into
the infrastructure layer, out of the application layer.
https://www.infoq.com/articles/linkerd-v2-production-adoption/
28
29. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Service Proxy Features
• Metrics without instrumenting apps
• Trace flow of requests across services
• One stable URI for each service
• Service discovery
• Monitor request latency
• Routing - A/B testing, green/blue deployments
• Circuit breaking
• Protocol translation (HTTP, gRPC, Kafka Protocol, etc.)
• Mutual TLS (mTLS)
• SSL Termination
• Integrate with 3rd party tools like Prometheus, Grafana,
Zipkin, etc.
• Much more…
Observability
“is by far the most important thing that a Proxy and the Service
Mesh provide in a distributed Microservice architecture!” Matt Klein
29
30. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Example: Lyft today with “envoy” Proxy
• 100% (!!!) communication coverage - Everything talks through Envoy Proxies
• à Make monitoring, debugging, firefighting as consistent as possible
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
30
31. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Kafka Connect
Kafka Cluster
CRM
Integration
Clients and Servers are Independent (including their Ops Teams)
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Control
Plane
31
32. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Example: Kafka + Confluent REST Proxy
Envoy
Proxy
I am using REST too!
Kafka? Never heard
of her.
I’m using REST
to talk to a
service
I’m proxying
REST.
And also
logging stuff
to Kafka
Confluent
REST Proxy
I support only
TCP!
HTTP
HTTP
32
33. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Example: Kafka + Envoy Kafka Protocol Filter
Envoy
Proxy
I am using REST too!
Kafka? Never heard
of her.
I’m using REST
to talk to a
service
I’m proxying
REST.
And also
logging stuff
to Kafka
HTTP
TCP
(Kafka Protocol)
33
34. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Implementation: Kafka + Istio @ Banzai Cloud
34
https://banzaicloud.com/blog/kafka-on-istio-performance/
35. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Implementation: Kafka + Istio
35
https://banzaicloud.com/blog/kafka-on-istio-performance/
36. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Implementation: Kafka + ksqlDB + Istio
36
https://banzaicloud.com/blog/supertubes-ksql/
37. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
(Potential) Features for
Kafka + Service Mesh Implementation
Protocol conversion from HTTP / gRPC
to Kafka
• Tap feature to dump to a Kafka stream
• Protocol parsing for observability
(stats, logging, and trace linking with
HTTP RPCs)
• Shadow requests to a Kafka stream
instead of HTTP / gRPC shadow
• Integrate with Kafka Connect and its
whole ecosystem of connectors
Validation of Events
• Serialization format
(JSON, Avro, Protobuf, etc.)
• Message schema
• Headers, attributes, etc.
Security
• SSL Termination
• Mutual TLS (mTLS)
• Authorization
Proxy features
• Dynamic Routing
• Rate limiting at both the L4 connection
and L7 message level
• Filter, add compression, …
• Automatic topic name conversion (e.g. for
canary release or blue/green deployment)
Monitoring and Tracing
• Request logs and stats
• Data lineage / audit log
• Audit log by taking request logs and
enriching them with the user info.
• Client specific metrics (Byte rate per
client id / per consumer groups,
versions of the client libraries,
consumer lag monitoring for the entire
data center)
37
38. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
38
39. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
40. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Apache Kafka as Infrastructure for ML
41. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams/
ksqlDB
Kafka Connect
Confluent REST Proxy
Confluent Schema Registry
Go/.NET/Python
Kafka Producer
ksqlDB
Python
Client
42. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Streaming Analytics for
Predictive Maintenance at Scale
42
IoT
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Car Sensors
Streaming Platform
Other Components
Real Time
Monitoring
System
All
Data
Critical
Data
Ingest
Data
Potential
Detect
43. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Streaming Analytics for
Predictive Maintenance at Scale
43
IoT
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Car Sensors
Streaming Platform
Analytics Platform
Other Components
Real Time
Monitoring
System
All
Data
Critical
Data
Ingest
Data
Potential
DetectAnalytics
Platform
Train
Analytic
Model
Data
Processing
Analytic
Model
Preprocess
Data
Consume
Data
Deploy
Analytic Model
44. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Preprocessing
with ksqlDB
44
SELECT car_id, event_id, car_model_id, sensor_input
FROM car_sensor c
LEFT JOIN car_models m ON c.car_model_id = m.car_model_id
WHERE m.car_model_type ='Audi_A8';
45. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model BModel A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
45
Model X
(at a later time)
46. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Confluent Tiered Storage for Kafka
46
(Only available in Confluent Platform)
47. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Local Predictions
Model Training
in the Cloud
Model Deployment
at the Edge
Analytic Model
Separation of
Model Training and Model Inference
47
48. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id,
detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
48
49. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
50. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cybersecurity
The threat is real!
Challenges
Stealing IP
DDoS
Ransomware / wiperware
WannaCry, NotPetya, …
Damage: Billions of dollars
”Supply chain attack”
Digital
Transformation
Networking
Communication
Connectivity
Open standards
”Always-on”
51. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Legacy SIEM needs to evolve
ForwarderNetwork traffic
Firewall logs
RDBMS
Application logs
Adaptors
Beats
Sensor Data
Challenges:
● Proprietary forwarders that can only
send data to single source
● Data locked from being shared
● Difficult to scale with growing data
volumes
● Prohibitively high indexing costs
● Unable to filter out noisy data
● Slow batch processing
HTTP proxy logs
52. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
AI/ML
Modernized security information and event management (SIEM)
Filter, transform
aggregate
APP SIEM Index
Search
Curated streams
Forensic
Archive
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
Payment Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
APP
Stateful
real-time analytics
53. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
54. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
55. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
How does
Confluent
help?
56. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
57. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid, Multi-
Region)
Build Contextual Event-
Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
58. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Confluent Platform
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF
CHOICE
COMMITTER-DRIVEN
EXPERTISE
PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based
Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete
Engagement Model
Revenue / Cost / Risk
Impact
TCO / ROI
Executive Buyer