The Rise of Event Streaming in Financial Services - Use Cases, Architectures and Examples powered by Apache Kafka.
The New FinServ Enterprise Reality: Every company is a software company. Innovate OR be Disrupted. Learn how Event Streaming with Apache Kafka and its ecosystem help...
More details:
https://www.kai-waehner.de/apache-kafka-financial-services-industry-banking-finserv-payment-fraud-middleware-messaging-transactions
https://www.kai-waehner.de/blog/2020/04/15/apache-kafka-machine-learning-banking-finance-industry/
https://www.kai-waehner.de/blog/2020/04/24/mainframe-offloading-replacement-apache-kafka-connect-ibm-db2-mq-cdc-cobol/
Apache Kafka in Financial Services - Use Cases and Architectures
1. The Rise of Event Streaming
in Financial Services
Use Cases, Architectures and Examples powered by Apache Kafka
Kai Waehner
Field CTO
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
2. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The world is changing.
3. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data
4. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Stores become Software
5. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Banks become Software
6. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Taxis become Software
2
min
7. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Cars become Software
8. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Car Production becomes Software
9. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
This transformation is
happening everywhere
10. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Transportation
13. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The New FinServ Enterprise Reality
- Innovate OR be Disrupted
New Fintech Entrants.
Tech Giants moving into FS e.g. Apple,
Alibaba, Google and Tencent..
Supermarket banking. Existing banks
opening parallel digital only banks. - Chatbots
- Robotics, AI, Machine Learning
- Biometrics - stronger security
- Blockchain for digital transactions
- Regulatory requirements
- Cyber Security
Omni-channel banking experiences
(online, branch, contact center etc)
Sharing Economy (peer-to-peer lending)
Decentralized asset ownership
Open Banking
Automation: Customer journey - Instant
end-to-end fulfillment of customer needs.
1. Create New
Business Models
2. Deliver new, real-
time customer
experiences
3. Deliver
massive internal
efficiencies
Automation - improved
service e.g. Fraud prevention
better customer 360
Fraud detection
Personalized recommendations
Real-time activity tracking and notifications
14. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The New FinServ Enterprise Reality
- Every company is a software company
● 10,000 of 40,000 employees are software
engineers
● 1.5B (billion!) lines of code across 7,000+
applications
● Employs over 50,000 people in technology
and has $10B+ technology spend.
15. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
What enables this
transformation?
16. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers
17. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Would you blindly cross the street with
traffic information that is 5 minutes old?
18. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Real-time Data beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases
19. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
This is a fundamental paradigm shift...
19
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
20. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
What is Event Streaming?
21. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Event Streaming is the
Central Nervous System
for today’s enterprises.
Apache Kafka®
is the technology.
22. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Banking
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka
23. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Shipping
Reporting
Orders
Frontend
Retail Example
24. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
25. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
ETL/Data Integration Messaging
Highly Scalable
Durable
Persistent
Ordered
Real-time Difficult to Scale
No Persistence After
Consumption
No Replay
Batch
Expensive
Time Consuming
26. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Highly Scalable
Persistent
ETL/Data Integration MessagingETL/Data Integration MessagingMessaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Real-timeHighly Scalable
Durable
Persistent
Ordered
Real-time
Event Streaming
27. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Global Scale
Real-time
Persistent Storage
Stream Processing
Data Integration
Apache Kafka
The De-facto Standard for Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache
Kafka
29. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Confluent pioneered Event Streaming
Hall of Innovation
CTO Innovation
Award Winner
2019
Enterprise Technology
Innovation
AWARDS
Confluent founders are
original creators of Kafka
Confluent team wrote
80% of Kafka software
commits, has over 1M hours
technical experience with
Kafka, and operates 5,000+
clusters
Confluent Cloud
is the only multi-cloud, fully
managed, pay-as-you-go
event streaming service in
the world
Confluent Platform
completes Apache Kafka
and turns it into a secure,
enterprise-ready platform
30. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
“We look at events as running our
business. Business people within
our organization want to be able to
react to events—and oftentimes it's
a combination of events.”
VP of Streaming Data Engineering
31. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
“Walmart is a $500 billion in revenue
company, so every second is worth
millions of dollars. Having Confluent
as our partner has been invaluable.
Kafka and Confluent are the
backbone of our digital
omnichannel transformation
and success at Walmart.”
VP of Walmart Cloud
32. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Cyber Security
High volume log ingest & anomaly
detection
Wealth Management
and Capital Markets
Trade Data Capture, Next Gen Pricing
Applications, Clearing & Settlement, OATS,
Next Gen Advisor workstations
Retail and Corp Banking
Fraud Detection, Credit analytics, Next
Generation Payment Hubs, Open Banking
Market & Credit Risk
Consolidates data across dozens of
disparate risk systems
IT Modernization
Mainframe off-load, “Bridge to the Cloud” ,
microservices
Customer Experience
Tailored offerings and alerts, omni
channel, increase digital engagement
within Retail & Corp Banking, Wealth
Management
Kafka across Financial Services
33. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
34. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Third largest corporation in the
Nordic region and one of the top
10 financial services companies in
Europe based on market
capitalisation
Use Cases
-Regulatory Compliance
-Messaging Modernization
Challenges
● MiFID II compliance requirements
● Breaking down silos in the bank to get an aggregated view of all
equity trades
● Legacy systems support limited throughput and ETL
Solution
● Centralized Confluent deployment to free data stuck in legacy systems
● JMS client offload data from legacy systems
● Dropped analytics turnaround time from weeks to instant reporting,
● Provided analysts access to trade data in real-time
● Reduced overall platform costs by 73%
Mainframe
APACHE KAFKA
Confluent
JMS
35. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
“… rescue data off of the mainframe, in a cloud native,
microservice-based fashion … [to] … significantly reduce the
reads on the mainframe, saving RBC fixed infrastructure
costs (OPEX). RBC stayed compliant with bank regulations
and business logic, and is now able to create new
applications using the same event-based architecture.”
Mainframe Offloading
for massive cost-savings
https://www.confluent.io/customers/rbc/
36. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
First pan-European exchange,
Euronext operates regulated
securities and derivatives markets
in Amsterdam, Brussels, Lisbon
and Paris, as well as a regulated
securities market in Ireland and
the UK.
Use Cases
-Market Data & Trade Platform
-Messaging Modernization
Challenges
● Develop a new trading platform for markets across multiple countries in EU
● Support high-volume, high-speed trading, provides clients with access to
real-time data.
● Mission-critical platform trading platform will support the market
capitalization of six European countries
Solution
● Developed a new event-driven trading platform, Optiq®, with tenfold increase
in capacity
● Average performance latency of 15ms for order round trip as well as for
market data
● Billions of messages per day with millisecond latencies
● Reliable 24/5 operations with dedicated enterprise support
● Development applications that use the Kafka Streams API library to perform
data enrichment in real time
“We have been very satisfied with Confluent Platform as the backbone of our persistence
engine. The platform has been super reliable. We have stringent requirements for real-time
performance and reliability, and we have confirmed - from proof-of-concept to deployment
of a cutting-edge production trading platform – that we made the right decision”
- Alain Courbebaisse, Chief Information Officer, Euronext
37. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
● Capital One “Second Look”
● Customers do not check
statements regularly
● Duplicate charges, high tips,
increased recurring charges
go unnoticed
● The right level of signal vs
noise for the consumer
● Preventing $150 of fraud on
average a year/customer
Use Cases:
-Customer 360
-Customer Notifications & Alerts
-Fraud Detection
38. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
What is Kafka?
39. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams
40. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka Loves Your Existing Systems.
...many more
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect
41. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka Loves Your Existing Systems.
✓ Hundreds of ready-to-use
connectors available
✓ Supported by Confluent
or by partners (certified)
✓ Managed Connectors in
Confluent Cloud
$ confluent-hub install mongodb/kafka-connect-mongodb:1.3.0
42. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Confluent Cloud
✓ Infinite Storage
Confluent Platform
✓ Tiered Storage (e.g., AWS S3)
Apache Kafka
• Tiered Storage upcoming
Kafka Stores Your Data Durably.
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1
43. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB
44. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Kafka Streams — #1 popular choice
✓ Client library for Java, Scala
✓ Elastic, highly scalable, fault-tolerant
✓ Filter, transform, join, aggregate, window, etc.
✓ Exactly-once processing
✓ Full Kafka security integration
Build Apps, not Processing Clusters
✓ Standard JVM application
✓ Run in container, VMs, K8S, bare-metal, etc.
✓ No separate ‘processing cluster’ needed
✓ For S, M, L, XXL use cases
Kafka Makes Your Business Real-time.
Kafka Cluster
REAL-TIME APPLICATIONS
45. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
ksqlDB, the Event Streaming Database
DB
APP
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
TABLES & STATE
ksqlDBDB One Solution for:
✓ capturing events
✓ stream processing
✓ serving both push
and pull queries
✓ also available in
Confluent Cloud
CREATE SOURCE CONNECTOR postgresSource WITH (...);
CREATE STREAM customer_changes …;
CREATE TABLE customers AS SELECT ... FROM customer_changes;
APP
“The easiest way to get started with Kafka.”
47. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.
48. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Distributed System with Replication and High Availability
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4
49. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Distributed System with Replication and High Availability
read,
write
App
(Kafka Streams)
Kafka
(data)
More Apps
(KSQL, Connect, Python,
REST, “You-name-it”)
BookingsTeam
FraudTeam
…
MobileTeam
…
50. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Global Central Nervous System
Your Business as Streams of Events, powered by Kafka
51. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Event Streaming Is The Future Of Data
51
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
52. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Cloud
Adoption
Journey from Mainframe
to Hybrid and Cloud
PHASE
3
Hybrid
Cloud
Cloud-First
Development
PHASE
2
PHASE
1
https://www.accenture.com/_acnmedia/pdf-70/accenture-moving-to-the-cloud-strategy-for-banks-in-north-america.pdf
Case Study - Bank CEO
“This is the last 5-year $20M IBM contract.
Get rid off the mainframe!”
53. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Strangler Design Pattern
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/
https://martinfowler.com/bliki/StranglerFigApplication.html
“The most important reason
to consider a strangler fig
application over a cut-over
rewrite is reduced risk.”
Martin Fowler
54. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Year 0: Direct Communication between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
55. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Year 1: Kafka for Decoupling between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd
Party CDC Tool
56. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Year 2 to 4: New Projects and Applications
Application
Kafka-native
Applications
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project
(Elastic, Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd
Party CDC Tool
57. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Year 5: Mainframe Replacement
Application
Big Data Project
(Elastic, Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
Core Banking ‘2020’
(Modern Technology)
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Kafka-native
Applications
Agile, Lightweight
(but Scalable, Robust)
Applications
58. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
59. Event Streaming with Apache Kafka– @KaiWaehner - www.kai-waehner.de
Why Confluent?
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE
COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema
Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete Engagement Model
Revenue / Cost / Risk Impact
TCO / ROI
Executive Buyer