The Top 5 Event Streaming Use Cases & Architectures in 2021

The Top 5 Event Streaming
Use Cases & Architectures in 2021
Hybrid Architectures, Edge Computing, Machine Learning, Cybersecurity, Service Mesh
Kai Waehner
Field CTO
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Gartner Top
Strategic Technology
Trends for 2021
https://www.gartner.com/smarterwithgartner/gartner-top-strategic-technology-trends-for-2021/

STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming

Global Scale
Real-time
Persistent Storage
Stream Processing
Data Integration
Apache Kafka
The De-facto Standard for Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache
Kafka

Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate
to Cloud
Mitigate
Risk (protect
money)
Key Drivers
Strategic
Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation &
improved in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous
System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital
One
Developer Velocity - Building
Stateful Financial Applications with
Kafka Streams: Funding Circle
Detect Fraud & Prevent Fraud in
Real Time: PayPal
Kafka as a Service - A Tale of
Security and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)

The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity

Why Kafka in Multiple Data Centers?
* Not a representative survey J
** ‘Many DCs’ does NOT necessarily mean more than one Kafka Cluster

Disaster Recovery – RPO and RTO
RPO = Recovery Point Objective
RTO = Recovery Time Objective

Disaster Recovery @ JPMorgan
https://www.confluent.io/kafka-summit-san-francisco-2019/secure-kafka-at-scale-in-true-multi-tenant-environment

Cluster Linking
• Hybrid-cloud and multi-cloud
• No additional infrastructure (such as Kafka Connect or MirrorMaker)
• Just configuration
• Regional or global

What is the right Hybrid Kafka Architecture for you?
(Hint: This is hard à Let’s guide you by our experts)
12
Latency
> 50ms
Latency
< 50ms
RTO = 0 RTO > 0 RPO = 0 RPO > 0 Single Region Multi-Region Global
Stretched Cluster
x x x x
Replicator
x x x x* x
Cluster-Linking
x x x x* x
MRC Sync
x x x x**
MRC Observer
x x x x**
* With a stretched cluster in a single region, you still have RTO & RPO = 0
** Requires 3 regions minimum

Kafka as a Service – Fully Managed?
Infrastructure
management
(commodity)
Scaling
● Upgrades (latest stable version of Kafka)
● Patching
● Maintenance
● Sizing (retention, latency, throughput, storage, etc.)
● Data balancing for optimal performance
● Performance tuning for real-time and latency requirements
● Fixing Kafka bugs
● Uptime monitoring and proactive remediation of issues
● Recovery support from data corruption
● Scaling the cluster as needed
● Data balancing the cluster as nodes are added
● Support for any Kafka issue with less than X minutes response time
Infra-as-a-Service
Harness full power of Kafka
Kafka-specific
management
Platform-as-a-Service
Evolve as you need
Future-proof
Mission-critical reliability
Most Kafka-as-a-Service offerings are partially-managed
Kafka as a Service should be a serverless experience with consumption-based pricing!

What is the “Edge” for Kafka?
• Edge is NOT a data center
• Kafka clients AND the Kafka broker(s)
• Offline business continuity
• Often 100+ locations
• Low-footprint and low-touch
• Hybrid integration
Example:
Single broker, 1 GB Ram
100 MB/sec

CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Customer
data
Train
schedule
Payment
data
Loyalty
information
Hybrid Architecture

Food
Inventory Loyalty
System
Traveler
Information
Orders Upsell to
first class
Customer
data
Train
schedule
Payment
data
Loyalty
information
Updated
SchedulesEvent Streaming
at the Edge

Data Processing
at the Edge
Time
P
C1
C2
C3
Know-your-customer
Loyalty app, predictive behavior, …
Estimated
time of arrival
Connect to the
gaming server
for kids
Play games, earn rewards, communicate
with other kids in the train, …
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency

Edge Kafka @ Royal Caribbean
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/

Edge Integration and Analytics @ WPX Energy
Edge processing and
replication to the cloud
in real-time at scale
in the oil&gas industry

Microservices to the rescue?
• Significant Operations Overhead
• Substantial DevOps Skills Required
• Implicit Interfaces
• Duplication Of Effort
http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
• Distributed System Complexity
• Asynchronicity Is Difficult
• Testability Challenges
22

Microservices can lead to Death-star Architectures
Netflix: https://www.slideshare.net/brucewong3/the-case-for-chaos
Twitter: https://twitter.com/adrianco/status/441883572618948608
Hail-o: http://www.sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-3/
450+ microservices 500+ microservices 500+ microservices
23

Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
24

Kafka Connect
Kafka Cluster
CRM
Integration
Domain-Driven Design (DDD) for your Microservice Architecture
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
25

•Decoupled
•Flexible
•Data Aware
•Elastic
26

Cloud-Native Deployment leveraging Kubernetes
27

Service Mesh
A microservice pattern to move visibility, reliability, and
security primitives for service-to-service communication into
the infrastructure layer, out of the application layer.
https://www.infoq.com/articles/linkerd-v2-production-adoption/
28

Service Proxy Features
• Metrics without instrumenting apps
• Trace flow of requests across services
• One stable URI for each service
• Service discovery
• Monitor request latency
• Routing - A/B testing, green/blue deployments
• Circuit breaking
• Protocol translation (HTTP, gRPC, Kafka Protocol, etc.)
• Mutual TLS (mTLS)
• SSL Termination
• Integrate with 3rd party tools like Prometheus, Grafana,
Zipkin, etc.
• Much more…
Observability
“is by far the most important thing that a Proxy and the Service
Mesh provide in a distributed Microservice architecture!” Matt Klein
29

Example: Lyft today with “envoy” Proxy
• 100% (!!!) communication coverage - Everything talks through Envoy Proxies
• à Make monitoring, debugging, firefighting as consistent as possible
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
30

Kafka Connect
Kafka Cluster
CRM
Integration
Clients and Servers are Independent (including their Ops Teams)
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Control
Plane
31

Example: Kafka + Confluent REST Proxy
Envoy
Proxy
I am using REST too!
Kafka? Never heard
of her.
I’m using REST
to talk to a
service
I’m proxying
REST.
And also
logging stuff
to Kafka
Confluent
REST Proxy
I support only
TCP!
HTTP
HTTP
32

Example: Kafka + Envoy Kafka Protocol Filter
Envoy
Proxy
I am using REST too!
Kafka? Never heard
of her.
I’m using REST
to talk to a
service
I’m proxying
REST.
And also
logging stuff
to Kafka
HTTP
TCP
(Kafka Protocol)
33

Implementation: Kafka + Istio @ Banzai Cloud
34
https://banzaicloud.com/blog/kafka-on-istio-performance/

Implementation: Kafka + Istio
35
https://banzaicloud.com/blog/kafka-on-istio-performance/

Implementation: Kafka + ksqlDB + Istio
36
https://banzaicloud.com/blog/supertubes-ksql/

(Potential) Features for
Kafka + Service Mesh Implementation
Protocol conversion from HTTP / gRPC
to Kafka
• Tap feature to dump to a Kafka stream
• Protocol parsing for observability
(stats, logging, and trace linking with
HTTP RPCs)
• Shadow requests to a Kafka stream
instead of HTTP / gRPC shadow
• Integrate with Kafka Connect and its
whole ecosystem of connectors
Validation of Events
• Serialization format
(JSON, Avro, Protobuf, etc.)
• Message schema
• Headers, attributes, etc.
Security
• SSL Termination
• Mutual TLS (mTLS)
• Authorization
Proxy features
• Dynamic Routing
• Rate limiting at both the L4 connection
and L7 message level
• Filter, add compression, …
• Automatic topic name conversion (e.g. for
canary release or blue/green deployment)
Monitoring and Tracing
• Request logs and stats
• Data lineage / audit log
• Audit log by taking request logs and
enriching them with the user info.
• Client specific metrics (Byte rate per
client id / per consumer groups,
versions of the client libraries,
consumer lag monitoring for the entire
data center)
37

•Decoupled
•Flexible
•Data Aware
•Elastic
38

Apache Kafka as Infrastructure for ML

Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams/
ksqlDB
Kafka Connect
Confluent REST Proxy
Confluent Schema Registry
Go/.NET/Python
Kafka Producer
ksqlDB
Python
Client

Streaming Analytics for
Predictive Maintenance at Scale
42
IoT
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Car Sensors
Streaming Platform
Other Components
Real Time
Monitoring
System
All
Data
Critical
Data
Ingest
Data
Potential
Detect

Streaming Analytics for
Predictive Maintenance at Scale
43
IoT
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Car Sensors
Streaming Platform
Analytics Platform
Other Components
Real Time
Monitoring
System
All
Data
Critical
Data
Ingest
Data
Potential
DetectAnalytics
Platform
Train
Analytic
Model
Data
Processing
Analytic
Model
Preprocess
Data
Consume
Data
Deploy
Analytic Model

Preprocessing
with ksqlDB
44
SELECT car_id, event_id, car_model_id, sensor_input
FROM car_sensor c
LEFT JOIN car_models m ON c.car_model_id = m.car_model_id
WHERE m.car_model_type ='Audi_A8';

Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model BModel A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
45
Model X
(at a later time)

Confluent Tiered Storage for Kafka
46
(Only available in Confluent Platform)

Local Predictions
Model Training
in the Cloud
Model Deployment
at the Edge
Analytic Model
Separation of
Model Training and Model Inference
47

“CREATE STREAM AnomalyDetection AS
SELECT sensor_id,
detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
48

Cybersecurity
The threat is real!
Challenges
Stealing IP
DDoS
Ransomware / wiperware
WannaCry, NotPetya, …
Damage: Billions of dollars
”Supply chain attack”
Digital
Transformation
Networking
Communication
Connectivity
Open standards
”Always-on”

Legacy SIEM needs to evolve
ForwarderNetwork traffic
Firewall logs
RDBMS
Application logs
Adaptors
Beats
Sensor Data
Challenges:
● Proprietary forwarders that can only
send data to single source
● Data locked from being shared
● Difficult to scale with growing data
volumes
● Prohibitively high indexing costs
● Unable to filter out noisy data
● Slow batch processing
HTTP proxy logs

AI/ML
Modernized security information and event management (SIEM)
Filter, transform
aggregate
APP SIEM Index
Search
Curated streams
Forensic
Archive
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
Payment Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
APP
Stateful
real-time analytics

Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html

Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html

How does
Confluent
help?

The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid, Multi-
Region)
Build Contextual Event-
Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...

Confluent Platform
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF
CHOICE
COMMITTER-DRIVEN
EXPERTISE
PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based
Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete
Engagement Model
Revenue / Cost / Risk
Impact
TCO / ROI
Executive Buyer

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

The Top 5 Event Streaming Use Cases & Architectures in 2021

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to The Top 5 Event Streaming Use Cases & Architectures in 2021

Similar to The Top 5 Event Streaming Use Cases & Architectures in 2021 (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

The Top 5 Event Streaming Use Cases & Architectures in 2021