SlideShare a Scribd company logo
1 of 59
The Top 5 Event Streaming
Use Cases & Architectures in 2021
Hybrid Architectures, Edge Computing, Machine Learning, Cybersecurity, Service Mesh
Kai Waehner
Field CTO
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Gartner Top
Strategic Technology
Trends for 2021
https://www.gartner.com/smarterwithgartner/gartner-top-strategic-technology-trends-for-2021/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Global Scale
Real-time
Persistent Storage
Stream Processing
Data Integration
Apache Kafka
The De-facto Standard for Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache
Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate
to Cloud
Mitigate
Risk (protect
money)
Key Drivers
Strategic
Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation &
improved in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous
System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital
One
Developer Velocity - Building
Stateful Financial Applications with
Kafka Streams: Funding Circle
Detect Fraud & Prevent Fraud in
Real Time: PayPal
Kafka as a Service - A Tale of
Security and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Why Kafka in Multiple Data Centers?
* Not a representative survey J
** ‘Many DCs’ does NOT necessarily mean more than one Kafka Cluster
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Disaster Recovery – RPO and RTO
RPO = Recovery Point Objective
RTO = Recovery Time Objective
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Disaster Recovery @ JPMorgan
https://www.confluent.io/kafka-summit-san-francisco-2019/secure-kafka-at-scale-in-true-multi-tenant-environment
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cluster Linking
• Hybrid-cloud and multi-cloud
• No additional infrastructure (such as Kafka Connect or MirrorMaker)
• Just configuration
• Regional or global
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
What is the right Hybrid Kafka Architecture for you?
(Hint: This is hard à Let’s guide you by our experts)
12
Latency
> 50ms
Latency
< 50ms
RTO = 0 RTO > 0 RPO = 0 RPO > 0 Single Region Multi-Region Global
Stretched Cluster
x x x x
Replicator
x x x x* x
Cluster-Linking
x x x x* x
MRC Sync
x x x x**
MRC Observer
x x x x**
* With a stretched cluster in a single region, you still have RTO & RPO = 0
** Requires 3 regions minimum
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Kafka as a Service – Fully Managed?
Infrastructure
management
(commodity)
Scaling
● Upgrades (latest stable version of Kafka)
● Patching
● Maintenance
● Sizing (retention, latency, throughput, storage, etc.)
● Data balancing for optimal performance
● Performance tuning for real-time and latency requirements
● Fixing Kafka bugs
● Uptime monitoring and proactive remediation of issues
● Recovery support from data corruption
● Scaling the cluster as needed
● Data balancing the cluster as nodes are added
● Support for any Kafka issue with less than X minutes response time
Infra-as-a-Service
Harness full power of Kafka
Kafka-specific
management
Platform-as-a-Service
Evolve as you need
Future-proof
Mission-critical reliability
Most Kafka-as-a-Service offerings are partially-managed
Kafka as a Service should be a serverless experience with consumption-based pricing!
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
What is the “Edge” for Kafka?
• Edge is NOT a data center
• Kafka clients AND the Kafka broker(s)
• Offline business continuity
• Often 100+ locations
• Low-footprint and low-touch
• Hybrid integration
Example:
Single broker, 1 GB Ram
100 MB/sec
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Hybrid Architecture
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Food
Inventory Loyalty
System
Traveler
Information
Orders Upsell to
first class
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Updated
SchedulesEvent Streaming
at the Edge
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Data Processing
at the Edge
Time
P
C1
C2
C3
Know-your-customer
Loyalty app, predictive behavior, …
Estimated
time of arrival
Connect to the
gaming server
for kids
Play games, earn rewards, communicate
with other kids in the train, …
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Edge Kafka @ Royal Caribbean
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Edge Integration and Analytics @ WPX Energy
Edge processing and
replication to the cloud
in real-time at scale
in the oil&gas industry
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Microservices to the rescue?
• Significant Operations Overhead
• Substantial DevOps Skills Required
• Implicit Interfaces
• Duplication Of Effort
http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
• Distributed System Complexity
• Asynchronicity Is Difficult
• Testability Challenges
22
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Microservices can lead to Death-star Architectures
Netflix: https://www.slideshare.net/brucewong3/the-case-for-chaos
Twitter: https://twitter.com/adrianco/status/441883572618948608
Hail-o: http://www.sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-3/
450+ microservices 500+ microservices 500+ microservices
23
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
24
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Kafka Connect
Kafka Cluster
CRM
Integration
Domain-Driven Design (DDD) for your Microservice Architecture
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
25
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
26
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cloud-Native Deployment leveraging Kubernetes
27
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Service Mesh
A microservice pattern to move visibility, reliability, and
security primitives for service-to-service communication into
the infrastructure layer, out of the application layer.
https://www.infoq.com/articles/linkerd-v2-production-adoption/
28
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Service Proxy Features
• Metrics without instrumenting apps
• Trace flow of requests across services
• One stable URI for each service
• Service discovery
• Monitor request latency
• Routing - A/B testing, green/blue deployments
• Circuit breaking
• Protocol translation (HTTP, gRPC, Kafka Protocol, etc.)
• Mutual TLS (mTLS)
• SSL Termination
• Integrate with 3rd party tools like Prometheus, Grafana,
Zipkin, etc.
• Much more…
Observability
“is by far the most important thing that a Proxy and the Service
Mesh provide in a distributed Microservice architecture!” Matt Klein
29
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Example: Lyft today with “envoy” Proxy
• 100% (!!!) communication coverage - Everything talks through Envoy Proxies
• à Make monitoring, debugging, firefighting as consistent as possible
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
30
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Kafka Connect
Kafka Cluster
CRM
Integration
Clients and Servers are Independent (including their Ops Teams)
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Control
Plane
31
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Example: Kafka + Confluent REST Proxy
Envoy
Proxy
I am using REST too!
Kafka? Never heard
of her.
I’m using REST
to talk to a
service
I’m proxying
REST.
And also
logging stuff
to Kafka
Confluent
REST Proxy
I support only
TCP!
HTTP
HTTP
32
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Example: Kafka + Envoy Kafka Protocol Filter
Envoy
Proxy
I am using REST too!
Kafka? Never heard
of her.
I’m using REST
to talk to a
service
I’m proxying
REST.
And also
logging stuff
to Kafka
HTTP
TCP
(Kafka Protocol)
33
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Implementation: Kafka + Istio @ Banzai Cloud
34
https://banzaicloud.com/blog/kafka-on-istio-performance/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Implementation: Kafka + Istio
35
https://banzaicloud.com/blog/kafka-on-istio-performance/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Implementation: Kafka + ksqlDB + Istio
36
https://banzaicloud.com/blog/supertubes-ksql/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
(Potential) Features for
Kafka + Service Mesh Implementation
Protocol conversion from HTTP / gRPC
to Kafka
• Tap feature to dump to a Kafka stream
• Protocol parsing for observability
(stats, logging, and trace linking with
HTTP RPCs)
• Shadow requests to a Kafka stream
instead of HTTP / gRPC shadow
• Integrate with Kafka Connect and its
whole ecosystem of connectors
Validation of Events
• Serialization format
(JSON, Avro, Protobuf, etc.)
• Message schema
• Headers, attributes, etc.
Security
• SSL Termination
• Mutual TLS (mTLS)
• Authorization
Proxy features
• Dynamic Routing
• Rate limiting at both the L4 connection
and L7 message level
• Filter, add compression, …
• Automatic topic name conversion (e.g. for
canary release or blue/green deployment)
Monitoring and Tracing
• Request logs and stats
• Data lineage / audit log
• Audit log by taking request logs and
enriching them with the user info.
• Client specific metrics (Byte rate per
client id / per consumer groups,
versions of the client libraries,
consumer lag monitoring for the entire
data center)
37
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
38
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Apache Kafka as Infrastructure for ML
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams/
ksqlDB
Kafka Connect
Confluent REST Proxy
Confluent Schema Registry
Go/.NET/Python
Kafka Producer
ksqlDB
Python
Client
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Streaming Analytics for
Predictive Maintenance at Scale
42
IoT
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Car Sensors
Streaming Platform
Other Components
Real Time
Monitoring
System
All
Data
Critical
Data
Ingest
Data
Potential
Detect
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Streaming Analytics for
Predictive Maintenance at Scale
43
IoT
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Car Sensors
Streaming Platform
Analytics Platform
Other Components
Real Time
Monitoring
System
All
Data
Critical
Data
Ingest
Data
Potential
DetectAnalytics
Platform
Train
Analytic
Model
Data
Processing
Analytic
Model
Preprocess
Data
Consume
Data
Deploy
Analytic Model
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Preprocessing
with ksqlDB
44
SELECT car_id, event_id, car_model_id, sensor_input
FROM car_sensor c
LEFT JOIN car_models m ON c.car_model_id = m.car_model_id
WHERE m.car_model_type ='Audi_A8';
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model BModel A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
45
Model X
(at a later time)
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Confluent Tiered Storage for Kafka
46
(Only available in Confluent Platform)
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Local Predictions
Model Training
in the Cloud
Model Deployment
at the Edge
Analytic Model
Separation of
Model Training and Model Inference
47
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id,
detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
48
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cybersecurity
The threat is real!
Challenges
Stealing IP
DDoS
Ransomware / wiperware
WannaCry, NotPetya, …
Damage: Billions of dollars
”Supply chain attack”
Digital
Transformation
Networking
Communication
Connectivity
Open standards
”Always-on”
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Legacy SIEM needs to evolve
ForwarderNetwork traffic
Firewall logs
RDBMS
Application logs
Adaptors
Beats
Sensor Data
Challenges:
● Proprietary forwarders that can only
send data to single source
● Data locked from being shared
● Difficult to scale with growing data
volumes
● Prohibitively high indexing costs
● Unable to filter out noisy data
● Slow batch processing
HTTP proxy logs
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
AI/ML
Modernized security information and event management (SIEM)
Filter, transform
aggregate
APP SIEM Index
Search
Curated streams
Forensic
Archive
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
Payment Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
APP
Stateful
real-time analytics
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
How does
Confluent
help?
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid, Multi-
Region)
Build Contextual Event-
Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Confluent Platform
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF
CHOICE
COMMITTER-DRIVEN
EXPERTISE
PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based
Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete
Engagement Model
Revenue / Cost / Risk
Impact
TCO / ROI
Executive Buyer
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

More Related Content

What's hot

Apache Kafka Streams + Machine Learning / Deep Learning
Apache Kafka Streams + Machine Learning / Deep LearningApache Kafka Streams + Machine Learning / Deep Learning
Apache Kafka Streams + Machine Learning / Deep LearningKai Wähner
 
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Araf Karsh Hamid
 
Top use cases for 2022 with Data in Motion and Apache Kafka
Top use cases for 2022 with Data in Motion and Apache KafkaTop use cases for 2022 with Data in Motion and Apache Kafka
Top use cases for 2022 with Data in Motion and Apache Kafkaconfluent
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023confluent
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaKai Wähner
 
Streaming all over the world Real life use cases with Kafka Streams
Streaming all over the world  Real life use cases with Kafka StreamsStreaming all over the world  Real life use cases with Kafka Streams
Streaming all over the world Real life use cases with Kafka Streamsconfluent
 
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWSBridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWSconfluent
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Servicesconfluent
 
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...Kai Wähner
 
IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X
IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X
IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X Kai Wähner
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
 
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Kai Wähner
 
Stream processing and managing real-time data
Stream processing and managing real-time dataStream processing and managing real-time data
Stream processing and managing real-time dataAmazon Web Services
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail IndustryKai Wähner
 
Building Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaBuilding Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaGuido Schmutz
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
 

What's hot (20)

Apache Kafka Streams + Machine Learning / Deep Learning
Apache Kafka Streams + Machine Learning / Deep LearningApache Kafka Streams + Machine Learning / Deep Learning
Apache Kafka Streams + Machine Learning / Deep Learning
 
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
 
Top use cases for 2022 with Data in Motion and Apache Kafka
Top use cases for 2022 with Data in Motion and Apache KafkaTop use cases for 2022 with Data in Motion and Apache Kafka
Top use cases for 2022 with Data in Motion and Apache Kafka
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache Kafka
 
Streaming all over the world Real life use cases with Kafka Streams
Streaming all over the world  Real life use cases with Kafka StreamsStreaming all over the world  Real life use cases with Kafka Streams
Streaming all over the world Real life use cases with Kafka Streams
 
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWSBridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Services
 
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
 
IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X
IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X
IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
 
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
 
Kafka internals
Kafka internalsKafka internals
Kafka internals
 
Stream processing and managing real-time data
Stream processing and managing real-time dataStream processing and managing real-time data
Stream processing and managing real-time data
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
 
Building Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaBuilding Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache Kafka
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
 

Similar to Top 5 Event Streaming Use Cases for 2021 with Apache Kafka

App modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent CloudApp modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent CloudKai Wähner
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesKai Wähner
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...HostedbyConfluent
 
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...confluent
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...confluent
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaKai Wähner
 
Beyond the Brokers: A Tour of the Kafka Ecosystem
Beyond the Brokers: A Tour of the Kafka EcosystemBeyond the Brokers: A Tour of the Kafka Ecosystem
Beyond the Brokers: A Tour of the Kafka Ecosystemconfluent
 
Beyond the brokers - A tour of the Kafka ecosystem
Beyond the brokers - A tour of the Kafka ecosystemBeyond the brokers - A tour of the Kafka ecosystem
Beyond the brokers - A tour of the Kafka ecosystemDamien Gasparina
 
Beyond the brokers - Un tour de l'écosystème Kafka
Beyond the brokers - Un tour de l'écosystème KafkaBeyond the brokers - Un tour de l'écosystème Kafka
Beyond the brokers - Un tour de l'écosystème KafkaFlorent Ramiere
 
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent RamièreAu delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramièreconfluent
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...Kai Wähner
 
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemBuilding event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemGuido Schmutz
 
IoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache KafkaIoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache Kafkaconfluent
 
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...Kai Wähner
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)Kai Wähner
 
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...Yoichi Kawasaki
 
Introduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - MadridIntroduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - MadridPaolo Castagna
 
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehnerNitin Kumar
 
Confluent Operator as Cloud-Native Kafka Operator for Kubernetes
Confluent Operator as Cloud-Native Kafka Operator for KubernetesConfluent Operator as Cloud-Native Kafka Operator for Kubernetes
Confluent Operator as Cloud-Native Kafka Operator for KubernetesKai Wähner
 

Similar to Top 5 Event Streaming Use Cases for 2021 with Apache Kafka (20)

App modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent CloudApp modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent Cloud
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid Cloud
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice Architectures
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
 
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache Kafka
 
Beyond the Brokers: A Tour of the Kafka Ecosystem
Beyond the Brokers: A Tour of the Kafka EcosystemBeyond the Brokers: A Tour of the Kafka Ecosystem
Beyond the Brokers: A Tour of the Kafka Ecosystem
 
Beyond the brokers - A tour of the Kafka ecosystem
Beyond the brokers - A tour of the Kafka ecosystemBeyond the brokers - A tour of the Kafka ecosystem
Beyond the brokers - A tour of the Kafka ecosystem
 
Beyond the brokers - Un tour de l'écosystème Kafka
Beyond the brokers - Un tour de l'écosystème KafkaBeyond the brokers - Un tour de l'écosystème Kafka
Beyond the brokers - Un tour de l'écosystème Kafka
 
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent RamièreAu delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
 
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemBuilding event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka Ecosystem
 
IoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache KafkaIoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache Kafka
 
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
 
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
 
Introduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - MadridIntroduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - Madrid
 
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
 
Confluent Operator as Cloud-Native Kafka Operator for Kubernetes
Confluent Operator as Cloud-Native Kafka Operator for KubernetesConfluent Operator as Cloud-Native Kafka Operator for Kubernetes
Confluent Operator as Cloud-Native Kafka Operator for Kubernetes
 

More from Kai Wähner

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Kai Wähner
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Kai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsKai Wähner
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationKai Wähner
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Kai Wähner
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaKai Wähner
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
 
Apache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryApache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryKai Wähner
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesKai Wähner
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaKai Wähner
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Kai Wähner
 

More from Kai Wähner (20)

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and Manufacturing
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and Logistics
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
 
Apache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryApache Kafka in the Insurance Industry
Apache Kafka in the Insurance Industry
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache Kafka
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?
 

Recently uploaded

Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 

Recently uploaded (20)

Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 

Top 5 Event Streaming Use Cases for 2021 with Apache Kafka

  • 1. The Top 5 Event Streaming Use Cases & Architectures in 2021 Hybrid Architectures, Edge Computing, Machine Learning, Cybersecurity, Service Mesh Kai Waehner Field CTO contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
  • 2. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Gartner Top Strategic Technology Trends for 2021 https://www.gartner.com/smarterwithgartner/gartner-top-strategic-technology-trends-for-2021/
  • 3. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC Event Streaming
  • 4. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Global Scale Real-time Persistent Storage Stream Processing Data Integration Apache Kafka The De-facto Standard for Real-Time Event Streaming Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka
  • 5. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $↔ Example Case Studies (of many)
  • 6. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 7. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 8. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Why Kafka in Multiple Data Centers? * Not a representative survey J ** ‘Many DCs’ does NOT necessarily mean more than one Kafka Cluster
  • 9. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Disaster Recovery – RPO and RTO RPO = Recovery Point Objective RTO = Recovery Time Objective
  • 10. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Disaster Recovery @ JPMorgan https://www.confluent.io/kafka-summit-san-francisco-2019/secure-kafka-at-scale-in-true-multi-tenant-environment
  • 11. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cluster Linking • Hybrid-cloud and multi-cloud • No additional infrastructure (such as Kafka Connect or MirrorMaker) • Just configuration • Regional or global
  • 12. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent What is the right Hybrid Kafka Architecture for you? (Hint: This is hard à Let’s guide you by our experts) 12 Latency > 50ms Latency < 50ms RTO = 0 RTO > 0 RPO = 0 RPO > 0 Single Region Multi-Region Global Stretched Cluster x x x x Replicator x x x x* x Cluster-Linking x x x x* x MRC Sync x x x x** MRC Observer x x x x** * With a stretched cluster in a single region, you still have RTO & RPO = 0 ** Requires 3 regions minimum
  • 13. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Kafka as a Service – Fully Managed? Infrastructure management (commodity) Scaling ● Upgrades (latest stable version of Kafka) ● Patching ● Maintenance ● Sizing (retention, latency, throughput, storage, etc.) ● Data balancing for optimal performance ● Performance tuning for real-time and latency requirements ● Fixing Kafka bugs ● Uptime monitoring and proactive remediation of issues ● Recovery support from data corruption ● Scaling the cluster as needed ● Data balancing the cluster as nodes are added ● Support for any Kafka issue with less than X minutes response time Infra-as-a-Service Harness full power of Kafka Kafka-specific management Platform-as-a-Service Evolve as you need Future-proof Mission-critical reliability Most Kafka-as-a-Service offerings are partially-managed Kafka as a Service should be a serverless experience with consumption-based pricing!
  • 14. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 15. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent What is the “Edge” for Kafka? • Edge is NOT a data center • Kafka clients AND the Kafka broker(s) • Offline business continuity • Often 100+ locations • Low-footprint and low-touch • Hybrid integration Example: Single broker, 1 GB Ram 100 MB/sec
  • 16. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent CRM 3rd party payment provider Context-specific real-time upsell Customer data Payment processing and fraud detection as a service Manager Get report API Customer Customer Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Hybrid Architecture
  • 17. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Food Inventory Loyalty System Traveler Information Orders Upsell to first class Customer data Train schedule Payment data Loyalty information Streams of real time events Updated SchedulesEvent Streaming at the Edge
  • 18. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Data Processing at the Edge Time P C1 C2 C3 Know-your-customer Loyalty app, predictive behavior, … Estimated time of arrival Connect to the gaming server for kids Play games, earn rewards, communicate with other kids in the train, … Always on (even “offline”) Replayability Reduced traffic cost Better latency
  • 19. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Edge Kafka @ Royal Caribbean https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
  • 20. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Edge Integration and Analytics @ WPX Energy Edge processing and replication to the cloud in real-time at scale in the oil&gas industry
  • 21. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 22. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Microservices to the rescue? • Significant Operations Overhead • Substantial DevOps Skills Required • Implicit Interfaces • Duplication Of Effort http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html • Distributed System Complexity • Asynchronicity Is Difficult • Testability Challenges 22
  • 23. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Microservices can lead to Death-star Architectures Netflix: https://www.slideshare.net/brucewong3/the-case-for-chaos Twitter: https://twitter.com/adrianco/status/441883572618948608 Hail-o: http://www.sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-3/ 450+ microservices 500+ microservices 500+ microservices 23
  • 24. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Key Requirements for Microservices •Decoupled •Flexible •Operationally Transparent •Data Aware •Elastic 24
  • 25. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Kafka Connect Kafka Cluster CRM Integration Domain-Driven Design (DDD) for your Microservice Architecture Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain è Independent and loosely coupled, but scalable, highly available and reliable! 25
  • 26. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Key Requirements for Microservices •Decoupled •Flexible •Operationally Transparent •Data Aware •Elastic 26
  • 27. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cloud-Native Deployment leveraging Kubernetes 27
  • 28. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Service Mesh A microservice pattern to move visibility, reliability, and security primitives for service-to-service communication into the infrastructure layer, out of the application layer. https://www.infoq.com/articles/linkerd-v2-production-adoption/ 28
  • 29. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Service Proxy Features • Metrics without instrumenting apps • Trace flow of requests across services • One stable URI for each service • Service discovery • Monitor request latency • Routing - A/B testing, green/blue deployments • Circuit breaking • Protocol translation (HTTP, gRPC, Kafka Protocol, etc.) • Mutual TLS (mTLS) • SSL Termination • Integrate with 3rd party tools like Prometheus, Grafana, Zipkin, etc. • Much more… Observability “is by far the most important thing that a Proxy and the Service Mesh provide in a distributed Microservice architecture!” Matt Klein 29
  • 30. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Example: Lyft today with “envoy” Proxy • 100% (!!!) communication coverage - Everything talks through Envoy Proxies • à Make monitoring, debugging, firefighting as consistent as possible https://www.youtube.com/watch?v=55yi4MMVBi4 Matt Klein at QCon NY 2018 30
  • 31. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Kafka Connect Kafka Cluster CRM Integration Clients and Servers are Independent (including their Ops Teams) Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain Proxy Proxy Proxy Proxy Proxy Proxy Control Plane 31
  • 32. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Example: Kafka + Confluent REST Proxy Envoy Proxy I am using REST too! Kafka? Never heard of her. I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka Confluent REST Proxy I support only TCP! HTTP HTTP 32
  • 33. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Example: Kafka + Envoy Kafka Protocol Filter Envoy Proxy I am using REST too! Kafka? Never heard of her. I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka HTTP TCP (Kafka Protocol) 33
  • 34. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Implementation: Kafka + Istio @ Banzai Cloud 34 https://banzaicloud.com/blog/kafka-on-istio-performance/
  • 35. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Implementation: Kafka + Istio 35 https://banzaicloud.com/blog/kafka-on-istio-performance/
  • 36. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Implementation: Kafka + ksqlDB + Istio 36 https://banzaicloud.com/blog/supertubes-ksql/
  • 37. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent (Potential) Features for Kafka + Service Mesh Implementation Protocol conversion from HTTP / gRPC to Kafka • Tap feature to dump to a Kafka stream • Protocol parsing for observability (stats, logging, and trace linking with HTTP RPCs) • Shadow requests to a Kafka stream instead of HTTP / gRPC shadow • Integrate with Kafka Connect and its whole ecosystem of connectors Validation of Events • Serialization format (JSON, Avro, Protobuf, etc.) • Message schema • Headers, attributes, etc. Security • SSL Termination • Mutual TLS (mTLS) • Authorization Proxy features • Dynamic Routing • Rate limiting at both the L4 connection and L7 message level • Filter, add compression, … • Automatic topic name conversion (e.g. for canary release or blue/green deployment) Monitoring and Tracing • Request logs and stats • Data lineage / audit log • Audit log by taking request logs and enriching them with the user info. • Client specific metrics (Byte rate per client id / per consumer groups, versions of the client libraries, consumer lag monitoring for the entire data center) 37
  • 38. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Key Requirements for Microservices •Decoupled •Flexible •Operationally Transparent •Data Aware •Elastic 38
  • 39. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 40. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Apache Kafka as Infrastructure for ML
  • 41. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Apache Kafka’s Open Ecosystem as Infrastructure for ML Kafka Streams/ ksqlDB Kafka Connect Confluent REST Proxy Confluent Schema Registry Go/.NET/Python Kafka Producer ksqlDB Python Client
  • 42. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Streaming Analytics for Predictive Maintenance at Scale 42 IoT Integration Layer Batch Analytics Platform BI Dashboard Streaming Platform Big Data Integration Layer Car Sensors Streaming Platform Other Components Real Time Monitoring System All Data Critical Data Ingest Data Potential Detect
  • 43. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Streaming Analytics for Predictive Maintenance at Scale 43 IoT Integration Layer Batch Analytics Platform BI Dashboard Streaming Platform Big Data Integration Layer Car Sensors Streaming Platform Analytics Platform Other Components Real Time Monitoring System All Data Critical Data Ingest Data Potential DetectAnalytics Platform Train Analytic Model Data Processing Analytic Model Preprocess Data Consume Data Deploy Analytic Model
  • 44. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Preprocessing with ksqlDB 44 SELECT car_id, event_id, car_model_id, sensor_input FROM car_sensor c LEFT JOIN car_models m ON c.car_model_id = m.car_model_id WHERE m.car_model_type ='Audi_A8';
  • 45. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Direct streaming ingestion for model training with TensorFlow I/O + Kafka Plugin (no additional data storage like S3 or HDFS required!) Time Model BModel A Producer Distributed Commit Log Streaming Ingestion and Model Training with TensorFlow IO https://github.com/tensorflow/io 45 Model X (at a later time)
  • 46. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Confluent Tiered Storage for Kafka 46 (Only available in Confluent Platform)
  • 47. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Local Predictions Model Training in the Cloud Model Deployment at the Edge Analytic Model Separation of Model Training and Model Inference 47
  • 48. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent “CREATE STREAM AnomalyDetection AS SELECT sensor_id, detectAnomaly(sensor_values) FROM car_engine;“ User Defined Function (UDF) Model Deployment with Apache Kafka, ksqlDB and TensorFlow 48
  • 49. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 50. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cybersecurity The threat is real! Challenges Stealing IP DDoS Ransomware / wiperware WannaCry, NotPetya, … Damage: Billions of dollars ”Supply chain attack” Digital Transformation Networking Communication Connectivity Open standards ”Always-on”
  • 51. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Legacy SIEM needs to evolve ForwarderNetwork traffic Firewall logs RDBMS Application logs Adaptors Beats Sensor Data Challenges: ● Proprietary forwarders that can only send data to single source ● Data locked from being shared ● Difficult to scale with growing data volumes ● Prohibitively high indexing costs ● Unable to filter out noisy data ● Slow batch processing HTTP proxy logs
  • 52. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent AI/ML Modernized security information and event management (SIEM) Filter, transform aggregate APP SIEM Index Search Curated streams Forensic Archive HDFS S3 Big Query CDC Syslog Network traffic Firewall logs RDBMS Application logs Payment Data HTTP proxy logs QRadar Arcsight Splunk Elastic APP Stateful real-time analytics
  • 53. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cyber Intelligence Platform leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more… https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
  • 54. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Fraud Detection at Scale in Real-Time for Billions of Messages https://www.infoq.com/presentations/paypal-data-service-fraud https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
  • 55. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent How does Confluent help?
  • 56. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Rise of Event Streaming 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 Companies trust and use Apache Kafka
  • 57. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi- Region) Build Contextual Event- Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management...
  • 58. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Confluent Platform Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional Services Enterprise Support Apache Kafka EFFICIENT OPERATIONS AT SCALE PRODUCTION- STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY SQL-based Stream Processing KSQL (ksqlDB) Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development non-Java clients | REST Proxy GUI-driven Mgmt & Monitoring Control Center Flexible DevOps Automation Operator | Ansible Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Enterprise-grade Security RBAC | Secrets | Audit logs Data Compatibility Schema Registry | Schema Validation Global Resilience Multi-Region Clusters | Replicator Developer Operator Architect Open Source | Community licensed PARTNERSHIP FOR BUSINESS SUCCESS Complete Engagement Model Revenue / Cost / Risk Impact TCO / ROI Executive Buyer