SlideShare a Scribd company logo
1 of 60
The Rise of Event Streaming
in Financial Services
Use Cases, Architectures and Examples powered by Apache Kafka
Kai Waehner
Field CTO
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The world is changing.
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Stores become Software
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Banks become Software
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Taxis become Software
2
min
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Cars become Software
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Car Production becomes Software
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
This transformation is
happening everywhere
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Transportation
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Retail
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Finance
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The New FinServ Enterprise Reality
- Innovate OR be Disrupted
New Fintech Entrants.
Tech Giants moving into FS e.g. Apple,
Alibaba, Google and Tencent..
Supermarket banking. Existing banks
opening parallel digital only banks. - Chatbots
- Robotics, AI, Machine Learning
- Biometrics - stronger security
- Blockchain for digital transactions
- Regulatory requirements
- Cyber Security
Omni-channel banking experiences
(online, branch, contact center etc)
Sharing Economy (peer-to-peer lending)
Decentralized asset ownership
Open Banking
Automation: Customer journey - Instant
end-to-end fulfillment of customer needs.
1. Create New
Business Models
2. Deliver new, real-
time customer
experiences
3. Deliver
massive internal
efficiencies
Automation - improved
service e.g. Fraud prevention
better customer 360
Fraud detection
Personalized recommendations
Real-time activity tracking and notifications
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The New FinServ Enterprise Reality
- Every company is a software company
● 10,000 of 40,000 employees are software
engineers
● 1.5B (billion!) lines of code across 7,000+
applications
● Employs over 50,000 people in technology
and has $10B+ technology spend.
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
What enables this
transformation?
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Would you blindly cross the street with
traffic information that is 5 minutes old?
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Real-time Data beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
This is a fundamental paradigm shift...
19
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
What is Event Streaming?
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Event Streaming is the
Central Nervous System
for today’s enterprises.
Apache Kafka®
is the technology.
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Banking
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Shipping
Reporting
Orders
Frontend
Retail Example
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
ETL/Data Integration Messaging
Highly Scalable
Durable
Persistent
Ordered
Real-time Difficult to Scale
No Persistence After
Consumption
No Replay
Batch
Expensive
Time Consuming
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Highly Scalable
Persistent
ETL/Data Integration MessagingETL/Data Integration MessagingMessaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Real-timeHighly Scalable
Durable
Persistent
Ordered
Real-time
Event Streaming
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Global Scale
Real-time
Persistent Storage
Stream Processing
Data Integration
Apache Kafka
The De-facto Standard for Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache
Kafka
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka & Confluent
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Confluent pioneered Event Streaming
Hall of Innovation
CTO Innovation
Award Winner
2019
Enterprise Technology
Innovation
AWARDS
Confluent founders are
original creators of Kafka
Confluent team wrote
80% of Kafka software
commits, has over 1M hours
technical experience with
Kafka, and operates 5,000+
clusters
Confluent Cloud
is the only multi-cloud, fully
managed, pay-as-you-go
event streaming service in
the world
Confluent Platform
completes Apache Kafka
and turns it into a secure,
enterprise-ready platform
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
“We look at events as running our
business. Business people within
our organization want to be able to
react to events—and oftentimes it's
a combination of events.”
VP of Streaming Data Engineering
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
“Walmart is a $500 billion in revenue
company, so every second is worth
millions of dollars. Having Confluent
as our partner has been invaluable.
Kafka and Confluent are the
backbone of our digital
omnichannel transformation
and success at Walmart.”
VP of Walmart Cloud
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Cyber Security
High volume log ingest & anomaly
detection
Wealth Management
and Capital Markets
Trade Data Capture, Next Gen Pricing
Applications, Clearing & Settlement, OATS,
Next Gen Advisor workstations
Retail and Corp Banking
Fraud Detection, Credit analytics, Next
Generation Payment Hubs, Open Banking
Market & Credit Risk
Consolidates data across dozens of
disparate risk systems
IT Modernization
Mainframe off-load, “Bridge to the Cloud” ,
microservices
Customer Experience
Tailored offerings and alerts, omni
channel, increase digital engagement
within Retail & Corp Banking, Wealth
Management
Kafka across Financial Services
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Third largest corporation in the
Nordic region and one of the top
10 financial services companies in
Europe based on market
capitalisation
Use Cases
-Regulatory Compliance
-Messaging Modernization
Challenges
● MiFID II compliance requirements
● Breaking down silos in the bank to get an aggregated view of all
equity trades
● Legacy systems support limited throughput and ETL
Solution
● Centralized Confluent deployment to free data stuck in legacy systems
● JMS client offload data from legacy systems
● Dropped analytics turnaround time from weeks to instant reporting,
● Provided analysts access to trade data in real-time
● Reduced overall platform costs by 73%
Mainframe
APACHE KAFKA
Confluent
JMS
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
“… rescue data off of the mainframe, in a cloud native,
microservice-based fashion … [to] … significantly reduce the
reads on the mainframe, saving RBC fixed infrastructure
costs (OPEX). RBC stayed compliant with bank regulations
and business logic, and is now able to create new
applications using the same event-based architecture.”
Mainframe Offloading
for massive cost-savings
https://www.confluent.io/customers/rbc/
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
First pan-European exchange,
Euronext operates regulated
securities and derivatives markets
in Amsterdam, Brussels, Lisbon
and Paris, as well as a regulated
securities market in Ireland and
the UK.
Use Cases
-Market Data & Trade Platform
-Messaging Modernization
Challenges
● Develop a new trading platform for markets across multiple countries in EU
● Support high-volume, high-speed trading, provides clients with access to
real-time data.
● Mission-critical platform trading platform will support the market
capitalization of six European countries
Solution
● Developed a new event-driven trading platform, Optiq®, with tenfold increase
in capacity
● Average performance latency of 15ms for order round trip as well as for
market data
● Billions of messages per day with millisecond latencies
● Reliable 24/5 operations with dedicated enterprise support
● Development applications that use the Kafka Streams API library to perform
data enrichment in real time
“We have been very satisfied with Confluent Platform as the backbone of our persistence
engine. The platform has been super reliable. We have stringent requirements for real-time
performance and reliability, and we have confirmed - from proof-of-concept to deployment
of a cutting-edge production trading platform – that we made the right decision”
- Alain Courbebaisse, Chief Information Officer, Euronext
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
● Capital One “Second Look”
● Customers do not check
statements regularly
● Duplicate charges, high tips,
increased recurring charges
go unnoticed
● The right level of signal vs
noise for the consumer
● Preventing $150 of fraud on
average a year/customer
Use Cases:
-Customer 360
-Customer Notifications & Alerts
-Fraud Detection
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
What is Kafka?
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka Loves Your Existing Systems.
...many more
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka Loves Your Existing Systems.
✓ Hundreds of ready-to-use
connectors available
✓ Supported by Confluent
or by partners (certified)
✓ Managed Connectors in
Confluent Cloud
$ confluent-hub install mongodb/kafka-connect-mongodb:1.3.0
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Confluent Cloud
✓ Infinite Storage
Confluent Platform
✓ Tiered Storage (e.g., AWS S3)
Apache Kafka
• Tiered Storage upcoming
Kafka Stores Your Data Durably.
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka Streams — #1 popular choice
✓ Client library for Java, Scala
✓ Elastic, highly scalable, fault-tolerant
✓ Filter, transform, join, aggregate, window, etc.
✓ Exactly-once processing
✓ Full Kafka security integration
Build Apps, not Processing Clusters
✓ Standard JVM application
✓ Run in container, VMs, K8S, bare-metal, etc.
✓ No separate ‘processing cluster’ needed
✓ For S, M, L, XXL use cases
Kafka Makes Your Business Real-time.
Kafka Cluster
REAL-TIME APPLICATIONS
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
ksqlDB, the Event Streaming Database
DB
APP
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
TABLES & STATE
ksqlDBDB One Solution for:
✓ capturing events
✓ stream processing
✓ serving both push
and pull queries
✓ also available in
Confluent Cloud
CREATE SOURCE CONNECTOR postgresSource WITH (...);
CREATE STREAM customer_changes …;
CREATE TABLE customers AS SELECT ... FROM customer_changes;
APP
“The easiest way to get started with Kafka.”
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Event Topics Storage PartitionsEvents / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
Event Topics Storage PartitionsEvents / sec Kafka Servers
250,000 500 25,000 25
Event Topics Storage PartitionsEvents / sec Kafka Servers
100 5 300 3
Kafka Scales with Your Business.
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Distributed System with Replication and High Availability
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Distributed System with Replication and High Availability
read,
write
App
(Kafka Streams)
Kafka
(data)
More Apps
(KSQL, Connect, Python,
REST, “You-name-it”)
BookingsTeam
FraudTeam
…
MobileTeam
…
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Global Central Nervous System
Your Business as Streams of Events, powered by Kafka
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Event Streaming Is The Future Of Data
51
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Cloud
Adoption
Journey from Mainframe
to Hybrid and Cloud
PHASE
3
Hybrid
Cloud
Cloud-First
Development
PHASE
2
PHASE
1
https://www.accenture.com/_acnmedia/pdf-70/accenture-moving-to-the-cloud-strategy-for-banks-in-north-america.pdf
Case Study - Bank CEO
“This is the last 5-year $20M IBM contract.
Get rid off the mainframe!”
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Strangler Design Pattern
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/
https://martinfowler.com/bliki/StranglerFigApplication.html
“The most important reason
to consider a strangler fig
application over a cut-over
rewrite is reduced risk.”
Martin Fowler
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Year 0: Direct Communication between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Year 1: Kafka for Decoupling between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd
Party CDC Tool
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Year 2 to 4: New Projects and Applications
Application
Kafka-native
Applications
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project
(Elastic, Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd
Party CDC Tool
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Year 5: Mainframe Replacement
Application
Big Data Project
(Elastic, Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
Core Banking ‘2020’
(Modern Technology)
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Kafka-native
Applications
Agile, Lightweight
(but Scalable, Robust)
Applications
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Why Confluent?
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE
COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema
Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete Engagement Model
Revenue / Cost / Risk Impact
TCO / ROI
Executive Buyer
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

More Related Content

What's hot

Stream processing and managing real-time data
Stream processing and managing real-time dataStream processing and managing real-time data
Stream processing and managing real-time dataAmazon Web Services
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023confluent
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner
 
Building Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaBuilding Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaGuido Schmutz
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesKai Wähner
 
Kafka and Machine Learning in Banking and Insurance Industry
Kafka and Machine Learning in Banking and Insurance IndustryKafka and Machine Learning in Banking and Insurance Industry
Kafka and Machine Learning in Banking and Insurance IndustryKai Wähner
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using KafkaKnoldus Inc.
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Servicesconfluent
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryKai Wähner
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
 
How to Quantify the Value of Kafka in Your Organization
How to Quantify the Value of Kafka in Your Organization How to Quantify the Value of Kafka in Your Organization
How to Quantify the Value of Kafka in Your Organization confluent
 
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
 
Benefits of Stream Processing and Apache Kafka Use Cases
Benefits of Stream Processing and Apache Kafka Use CasesBenefits of Stream Processing and Apache Kafka Use Cases
Benefits of Stream Processing and Apache Kafka Use Casesconfluent
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaKai Wähner
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
 

What's hot (20)

Stream processing and managing real-time data
Stream processing and managing real-time dataStream processing and managing real-time data
Stream processing and managing real-time data
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
 
Building Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaBuilding Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache Kafka
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice Architectures
 
Kafka internals
Kafka internalsKafka internals
Kafka internals
 
Kafka and Machine Learning in Banking and Insurance Industry
Kafka and Machine Learning in Banking and Insurance IndustryKafka and Machine Learning in Banking and Insurance Industry
Kafka and Machine Learning in Banking and Insurance Industry
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database System
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using Kafka
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Services
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel Industry
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
 
How to Quantify the Value of Kafka in Your Organization
How to Quantify the Value of Kafka in Your Organization How to Quantify the Value of Kafka in Your Organization
How to Quantify the Value of Kafka in Your Organization
 
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
 
Benefits of Stream Processing and Apache Kafka Use Cases
Benefits of Stream Processing and Apache Kafka Use CasesBenefits of Stream Processing and Apache Kafka Use Cases
Benefits of Stream Processing and Apache Kafka Use Cases
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache Kafka
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 

Similar to Apache Kafka in Financial Services - Use Cases and Architectures

Event Streaming in Retail with Apache Kafka
Event Streaming in Retail with Apache KafkaEvent Streaming in Retail with Apache Kafka
Event Streaming in Retail with Apache KafkaKai Wähner
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsKai Wähner
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail IndustryKai Wähner
 
Apache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionApache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionKai Wähner
 
Apache Kafka® and Analytics in a Connected IoT World
Apache Kafka® and Analytics in a Connected IoT WorldApache Kafka® and Analytics in a Connected IoT World
Apache Kafka® and Analytics in a Connected IoT Worldconfluent
 
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertFast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertconfluent
 
Apache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart CityApache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart CityKai Wähner
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaKai Wähner
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
 
Why Build an Apache Kafka® Connector
Why Build an Apache Kafka® ConnectorWhy Build an Apache Kafka® Connector
Why Build an Apache Kafka® Connectorconfluent
 
Mit Streaming die Brücken zum Erfolg bauen
Mit Streaming die Brücken zum Erfolg bauenMit Streaming die Brücken zum Erfolg bauen
Mit Streaming die Brücken zum Erfolg bauenconfluent
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...Kai Wähner
 
IoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache KafkaIoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache Kafkaconfluent
 
Webinar-Reihe: Realtime Retail Snack DCCS & Confluent 27. September 2022
Webinar-Reihe: Realtime Retail Snack DCCS & Confluent 27. September 2022Webinar-Reihe: Realtime Retail Snack DCCS & Confluent 27. September 2022
Webinar-Reihe: Realtime Retail Snack DCCS & Confluent 27. September 2022confluent
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Kai Wähner
 
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)Kai Wähner
 
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehnerNitin Kumar
 
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...Kai Wähner
 
Event Streaming in the Telco Industry with Apache Kafka® and Confluent
Event Streaming in the Telco Industry with Apache Kafka® and ConfluentEvent Streaming in the Telco Industry with Apache Kafka® and Confluent
Event Streaming in the Telco Industry with Apache Kafka® and Confluentconfluent
 
Set Your Data In Motion - CTO Roundtable
Set Your Data In Motion - CTO RoundtableSet Your Data In Motion - CTO Roundtable
Set Your Data In Motion - CTO Roundtableconfluent
 

Similar to Apache Kafka in Financial Services - Use Cases and Architectures (20)

Event Streaming in Retail with Apache Kafka
Event Streaming in Retail with Apache KafkaEvent Streaming in Retail with Apache Kafka
Event Streaming in Retail with Apache Kafka
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and Logistics
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
 
Apache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionApache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy Production
 
Apache Kafka® and Analytics in a Connected IoT World
Apache Kafka® and Analytics in a Connected IoT WorldApache Kafka® and Analytics in a Connected IoT World
Apache Kafka® and Analytics in a Connected IoT World
 
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertFast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
 
Apache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart CityApache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart City
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache Kafka
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
 
Why Build an Apache Kafka® Connector
Why Build an Apache Kafka® ConnectorWhy Build an Apache Kafka® Connector
Why Build an Apache Kafka® Connector
 
Mit Streaming die Brücken zum Erfolg bauen
Mit Streaming die Brücken zum Erfolg bauenMit Streaming die Brücken zum Erfolg bauen
Mit Streaming die Brücken zum Erfolg bauen
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
 
IoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache KafkaIoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache Kafka
 
Webinar-Reihe: Realtime Retail Snack DCCS & Confluent 27. September 2022
Webinar-Reihe: Realtime Retail Snack DCCS & Confluent 27. September 2022Webinar-Reihe: Realtime Retail Snack DCCS & Confluent 27. September 2022
Webinar-Reihe: Realtime Retail Snack DCCS & Confluent 27. September 2022
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
 
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
 
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
 
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
Apache Kafka in the Telco Industry (OSS, BSS, OTT, IMS, NFV, Middleware, Main...
 
Event Streaming in the Telco Industry with Apache Kafka® and Confluent
Event Streaming in the Telco Industry with Apache Kafka® and ConfluentEvent Streaming in the Telco Industry with Apache Kafka® and Confluent
Event Streaming in the Telco Industry with Apache Kafka® and Confluent
 
Set Your Data In Motion - CTO Roundtable
Set Your Data In Motion - CTO RoundtableSet Your Data In Motion - CTO Roundtable
Set Your Data In Motion - CTO Roundtable
 

More from Kai Wähner

Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Kai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Kai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationKai Wähner
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaKai Wähner
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesKai Wähner
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaKai Wähner
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Kai Wähner
 

More from Kai Wähner (16)

Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid Cloud
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and Manufacturing
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache Kafka
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?
 

Recently uploaded

About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9Jürgen Gutsch
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example ProjectMastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example Projectwajrcs
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Kubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptxKubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptxPrakarsh -
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Neo4j
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 

Recently uploaded (20)

About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example ProjectMastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Kubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptxKubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptx
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Sustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire ThornewillSustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire Thornewill
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Program with GUTs
Program with GUTsProgram with GUTs
Program with GUTs
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 

Apache Kafka in Financial Services - Use Cases and Architectures

  • 1. The Rise of Event Streaming in Financial Services Use Cases, Architectures and Examples powered by Apache Kafka Kai Waehner Field CTO contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
  • 2. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de The world is changing.
  • 3. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de The New Business Reality Technology is the business Innovation required for survival Yesterday’s data = failure Modern, real-time data infrastructure is required. Technology was a support function Innovation required for growth “Good enough” to run on yesterday’s data
  • 4. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Stores become Software
  • 5. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Banks become Software
  • 6. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Taxis become Software 2 min
  • 7. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Cars become Software
  • 8. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Car Production becomes Software
  • 9. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de This transformation is happening everywhere
  • 10. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Transportation
  • 11. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Retail
  • 12. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Finance
  • 13. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de The New FinServ Enterprise Reality - Innovate OR be Disrupted New Fintech Entrants. Tech Giants moving into FS e.g. Apple, Alibaba, Google and Tencent.. Supermarket banking. Existing banks opening parallel digital only banks. - Chatbots - Robotics, AI, Machine Learning - Biometrics - stronger security - Blockchain for digital transactions - Regulatory requirements - Cyber Security Omni-channel banking experiences (online, branch, contact center etc) Sharing Economy (peer-to-peer lending) Decentralized asset ownership Open Banking Automation: Customer journey - Instant end-to-end fulfillment of customer needs. 1. Create New Business Models 2. Deliver new, real- time customer experiences 3. Deliver massive internal efficiencies Automation - improved service e.g. Fraud prevention better customer 360 Fraud detection Personalized recommendations Real-time activity tracking and notifications
  • 14. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de The New FinServ Enterprise Reality - Every company is a software company ● 10,000 of 40,000 employees are software engineers ● 1.5B (billion!) lines of code across 7,000+ applications ● Employs over 50,000 people in technology and has $10B+ technology spend.
  • 15. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de What enables this transformation?
  • 16. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Cloud Machine Learning Mobile Event Streaming Rethink Decision Making Rethink User Experience Rethink Data Rethink Data Centers
  • 17. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Would you blindly cross the street with traffic information that is 5 minutes old?
  • 18. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Real-time Data beats Slow Data. Transportation Real-time sensor diagnostics Driver-rider match ETA updates Banking Fraud detection Trading, risk systems Mobile applications / customer experience Retail Real-time inventory Real-time POS reporting Personalization Entertainment Real-time recommendations Personalized news feed In-app purchases
  • 19. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de This is a fundamental paradigm shift... 19 Infrastructure as code Data as continuous streams of events Future of the datacenter Future of data Cloud Event Streaming
  • 20. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de What is Event Streaming?
  • 21. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Event Streaming is the Central Nervous System for today’s enterprises. Apache Kafka® is the technology.
  • 22. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de ‘Event’ is what happens in your business Transportation TPMS sensor in Carol’s car detected low tire-pressure at 5:11am. Kafka Banking Alice sent $250 to Bob on Friday at 7:34pm. Kafka Retail Sabine’s order of a Fujifilm camera was shipped at 9:10am. Kafka
  • 23. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Central Nervous System Your Business as Streams of Events, powered by Kafka Inventory Shipping Reporting Orders Frontend Retail Example
  • 24. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de The Rise of Event Streaming 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 Companies trust and use Apache Kafka
  • 25. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de ETL/Data Integration Messaging Highly Scalable Durable Persistent Ordered Real-time Difficult to Scale No Persistence After Consumption No Replay Batch Expensive Time Consuming
  • 26. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Highly Scalable Persistent ETL/Data Integration MessagingETL/Data Integration MessagingMessaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Real-timeHighly Scalable Durable Persistent Ordered Real-time Event Streaming
  • 27. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Global Scale Real-time Persistent Storage Stream Processing Data Integration Apache Kafka The De-facto Standard for Real-Time Event Streaming Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka
  • 28. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Kafka & Confluent
  • 29. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Confluent pioneered Event Streaming Hall of Innovation CTO Innovation Award Winner 2019 Enterprise Technology Innovation AWARDS Confluent founders are original creators of Kafka Confluent team wrote 80% of Kafka software commits, has over 1M hours technical experience with Kafka, and operates 5,000+ clusters Confluent Cloud is the only multi-cloud, fully managed, pay-as-you-go event streaming service in the world Confluent Platform completes Apache Kafka and turns it into a secure, enterprise-ready platform
  • 30. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de “We look at events as running our business. Business people within our organization want to be able to react to events—and oftentimes it's a combination of events.” VP of Streaming Data Engineering
  • 31. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de “Walmart is a $500 billion in revenue company, so every second is worth millions of dollars. Having Confluent as our partner has been invaluable. Kafka and Confluent are the backbone of our digital omnichannel transformation and success at Walmart.” VP of Walmart Cloud
  • 32. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Cyber Security High volume log ingest & anomaly detection Wealth Management and Capital Markets Trade Data Capture, Next Gen Pricing Applications, Clearing & Settlement, OATS, Next Gen Advisor workstations Retail and Corp Banking Fraud Detection, Credit analytics, Next Generation Payment Hubs, Open Banking Market & Credit Risk Consolidates data across dozens of disparate risk systems IT Modernization Mainframe off-load, “Bridge to the Cloud” , microservices Customer Experience Tailored offerings and alerts, omni channel, increase digital engagement within Retail & Corp Banking, Wealth Management Kafka across Financial Services
  • 33. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Fraud Detection at Scale in Real-Time for Billions of Messages https://www.infoq.com/presentations/paypal-data-service-fraud https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
  • 34. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Third largest corporation in the Nordic region and one of the top 10 financial services companies in Europe based on market capitalisation Use Cases -Regulatory Compliance -Messaging Modernization Challenges ● MiFID II compliance requirements ● Breaking down silos in the bank to get an aggregated view of all equity trades ● Legacy systems support limited throughput and ETL Solution ● Centralized Confluent deployment to free data stuck in legacy systems ● JMS client offload data from legacy systems ● Dropped analytics turnaround time from weeks to instant reporting, ● Provided analysts access to trade data in real-time ● Reduced overall platform costs by 73% Mainframe APACHE KAFKA Confluent JMS
  • 35. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de “… rescue data off of the mainframe, in a cloud native, microservice-based fashion … [to] … significantly reduce the reads on the mainframe, saving RBC fixed infrastructure costs (OPEX). RBC stayed compliant with bank regulations and business logic, and is now able to create new applications using the same event-based architecture.” Mainframe Offloading for massive cost-savings https://www.confluent.io/customers/rbc/
  • 36. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de First pan-European exchange, Euronext operates regulated securities and derivatives markets in Amsterdam, Brussels, Lisbon and Paris, as well as a regulated securities market in Ireland and the UK. Use Cases -Market Data & Trade Platform -Messaging Modernization Challenges ● Develop a new trading platform for markets across multiple countries in EU ● Support high-volume, high-speed trading, provides clients with access to real-time data. ● Mission-critical platform trading platform will support the market capitalization of six European countries Solution ● Developed a new event-driven trading platform, Optiq®, with tenfold increase in capacity ● Average performance latency of 15ms for order round trip as well as for market data ● Billions of messages per day with millisecond latencies ● Reliable 24/5 operations with dedicated enterprise support ● Development applications that use the Kafka Streams API library to perform data enrichment in real time “We have been very satisfied with Confluent Platform as the backbone of our persistence engine. The platform has been super reliable. We have stringent requirements for real-time performance and reliability, and we have confirmed - from proof-of-concept to deployment of a cutting-edge production trading platform – that we made the right decision” - Alain Courbebaisse, Chief Information Officer, Euronext
  • 37. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de ● Capital One “Second Look” ● Customers do not check statements regularly ● Duplicate charges, high tips, increased recurring charges go unnoticed ● The right level of signal vs noise for the consumer ● Preventing $150 of fraud on average a year/customer Use Cases: -Customer 360 -Customer Notifications & Alerts -Fraud Detection
  • 38. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de What is Kafka?
  • 39. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Kafka: The Trinity of Event Streaming 01 Publish & Subscribe to Streams of Events 02 Store your Event Streams 03 Process & Analyze your Events Streams
  • 40. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Kafka Loves Your Existing Systems. ...many more Other Systems Other Systems Kafka Connect Kafka Cluster Kafka Connect
  • 41. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Kafka Loves Your Existing Systems. ✓ Hundreds of ready-to-use connectors available ✓ Supported by Confluent or by partners (certified) ✓ Managed Connectors in Confluent Cloud $ confluent-hub install mongodb/kafka-connect-mongodb:1.3.0
  • 42. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Confluent Cloud ✓ Infinite Storage Confluent Platform ✓ Tiered Storage (e.g., AWS S3) Apache Kafka • Tiered Storage upcoming Kafka Stores Your Data Durably. Kafka is the source of truth. Powers NYTimes.com, and stores all articles ever published since 1851. September 30, 1851, Page 1
  • 43. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Kafka Makes Your Business Real-time. CREATE STREAM payments (user VARCHAR, amount INT) WITH (kafka_topic = 'all_payments', value_format = 'avro'); CREDIT SERVICE ksqlDB CREATE TABLE credit_scores AS SELECT user, updateScore(p.amount) AS credit_score FROM payments AS p GROUP BY user EMIT CHANGES; RISK SERVICE ksqlDB
  • 44. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Kafka Streams — #1 popular choice ✓ Client library for Java, Scala ✓ Elastic, highly scalable, fault-tolerant ✓ Filter, transform, join, aggregate, window, etc. ✓ Exactly-once processing ✓ Full Kafka security integration Build Apps, not Processing Clusters ✓ Standard JVM application ✓ Run in container, VMs, K8S, bare-metal, etc. ✓ No separate ‘processing cluster’ needed ✓ For S, M, L, XXL use cases Kafka Makes Your Business Real-time. Kafka Cluster REAL-TIME APPLICATIONS
  • 45. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de ksqlDB, the Event Streaming Database DB APP APP PULL PUSH CONNECTORS STREAM PROCESSING TABLES & STATE ksqlDBDB One Solution for: ✓ capturing events ✓ stream processing ✓ serving both push and pull queries ✓ also available in Confluent Cloud CREATE SOURCE CONNECTOR postgresSource WITH (...); CREATE STREAM customer_changes …; CREATE TABLE customers AS SELECT ... FROM customer_changes; APP “The easiest way to get started with Kafka.”
  • 46. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Event Topics Storage PartitionsEvents / sec Kafka Servers 10,000,000 25,000 1,000,000 1,500 Event Topics Storage PartitionsEvents / sec Kafka Servers 250,000 500 25,000 25 Event Topics Storage PartitionsEvents / sec Kafka Servers 100 5 300 3 Kafka Scales with Your Business.
  • 47. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Central Nervous System Your Business as Streams of Events, powered by Kafka Inventory Event streams are stored for reuse and with high availability. Shipping Events are processed in real- time as soon as they happen. Frontend Reporting Add new use cases easily by tapping into existing streams. Orders Event-driven apps and services communicate through streams.
  • 48. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Distributed System with Replication and High Availability Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
  • 49. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Distributed System with Replication and High Availability read, write App (Kafka Streams) Kafka (data) More Apps (KSQL, Connect, Python, REST, “You-name-it”) BookingsTeam FraudTeam … MobileTeam …
  • 50. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Global Central Nervous System Your Business as Streams of Events, powered by Kafka
  • 51. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Event Streaming Is The Future Of Data 51 Infrastructure as code Data as continuous streams of events Future of the datacenter Future of data Cloud Event Streaming
  • 52. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Cloud Adoption Journey from Mainframe to Hybrid and Cloud PHASE 3 Hybrid Cloud Cloud-First Development PHASE 2 PHASE 1 https://www.accenture.com/_acnmedia/pdf-70/accenture-moving-to-the-cloud-strategy-for-banks-in-north-america.pdf Case Study - Bank CEO “This is the last 5-year $20M IBM contract. Get rid off the mainframe!”
  • 53. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Strangler Design Pattern https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/ https://martinfowler.com/bliki/StranglerFigApplication.html “The most important reason to consider a strangler fig application over a cut-over rewrite is reduced risk.” Martin Fowler
  • 54. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Year 0: Direct Communication between Mainframe and App Application 1) Direct Legacy Mainframe Communication to App Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Core Banking ‘1970’ (Mainframe)
  • 55. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Year 1: Kafka for Decoupling between Mainframe and App Application 1) Direct Legacy Mainframe Communication to App 2) Kafka for Decoupling between Mainframe and App Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Core Banking ‘1970’ (Mainframe) Mainframe Integration - Change Data Capture (IIDR) - Kafka Connect (JMS, MQ, JDBC) - REST Proxy - Kafka Client - 3rd Party CDC Tool
  • 56. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Year 2 to 4: New Projects and Applications Application Kafka-native Applications Agile, Lightweight (but Scalable, Robust) Applications Big Data Project (Elastic, Spark, AWS Services, …) 1) Direct Legacy Mainframe Communication to App 2) Kafka for Decoupling between Mainframe and App 3) New Projects and Applications External Solution Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Core Banking ‘1970’ (Mainframe) Mainframe Integration - Change Data Capture (IIDR) - Kafka Connect (JMS, MQ, JDBC) - REST Proxy - Kafka Client - 3rd Party CDC Tool
  • 57. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Year 5: Mainframe Replacement Application Big Data Project (Elastic, Spark, AWS Services, …) 1) Direct Legacy Mainframe Communication to App 2) Kafka for Decoupling between Mainframe and App 3) New Projects and Applications 4) Mainframe Replacement External Solution Core Banking ‘2020’ (Modern Technology) Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Kafka-native Applications Agile, Lightweight (but Scalable, Robust) Applications
  • 58. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi-Region) Build Contextual Event-Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management...
  • 59. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de Why Confluent? Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional Services Enterprise Support Apache Kafka EFFICIENT OPERATIONS AT SCALE PRODUCTION- STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY SQL-based Stream Processing KSQL (ksqlDB) Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development non-Java clients | REST Proxy GUI-driven Mgmt & Monitoring Control Center Flexible DevOps Automation Operator | Ansible Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Enterprise-grade Security RBAC | Secrets | Audit logs Data Compatibility Schema Registry | Schema Validation Global Resilience Multi-Region Clusters | Replicator Developer Operator Architect Open Source | Community licensed PARTNERSHIP FOR BUSINESS SUCCESS Complete Engagement Model Revenue / Cost / Risk Impact TCO / ROI Executive Buyer