The rise of data in motion in the insurance industry is visible across all lines of business including life, healthcare, travel, vehicle, and others. Apache Kafka changes how enterprises rethink data. This blog post explores use cases and architectures for event streaming. Real-world examples from Generali, Centene, Humana, and Telsa show innovative insurance-related data integration and stream processing in real-time.

1. The Rise of Data in Motion in the Insurance Industry
Use Cases, Architectures and Examples powered by Apache Kafka
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de

2. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
The world is changing.

3. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
This transformation is
happening everywhere

4. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Stores become Software

5. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Banks become Software

6. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Taxis become Software
2
min

7. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Cars become Software

8. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Insurance becomes Software

9. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
What enables this
transformation?

10. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Challenges in the Insurance Industry
• Challenging market environments
• Stagnating economy
• Regulatory pressure
• Changes in customer expectations
• Proprietary and monolithic legacy applications
• Emerging competition from innovative insurtechs
• Emerging competition from other verticals that add
insurance products

11. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

12. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
This is a fundamental paradigm shift...
12
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

13. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Real-time Data beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Insurance
Claim processing
Claim fraud detection
Omnichannel
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

14. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
What is Event Streaming?

15. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Insurance
Alice filed an insurance claim Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka

16. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
An Event Streaming Platform is the
Underpinning of an Event-driven Architecture
16
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams of real time events
Stream processing
apps
Connectors
Connectors
Stream processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order

17. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
STREAM
PROCESSING
CONNECTORS
Example Architecture for Event Streaming
ksqlDB
KStreams
Real-time decision making for claim processing and fraud detection
Dashboard
Oracle
DB
Oracle
CDC
CONNECTOR
Salesforce CDC
CONNECTOR
Salesforce
Source / Sink
CONNECTOR
Fraud Detection App

18. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

19. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Real World Deployments

20. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Generali – Kafka as Integration Platform
for legacy and modern technologies
https://www.jug.ch/events/slides/190918_Microservices_and_Kafka_on_OpenShift.pdf

21. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Integration Platform
for legacy and modern technologies

22. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Database Integration and Offloading
Brownfield instead of Greenfield
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
DB Workload
DB Workload
DB Workload
Read
No DB Workload

23. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
DB Workload
DB Workload
DB Workload
Read
No DB Workload
Database Replacement
Brownfield instead of Greenfield

24. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Centene – Integration and Data Processing at Scale in Real-Time
Healthcare Insurer acts as intermediary for both government-sponsored and privately insured health care programs
Largest Medicaid and Medicare Managed Care Provider in the US
24
https://www.confluent.io/online-talks/building-an-enterprise-eventing-framework-on-demand/

25. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Swiss Mobiliar – Decoupling and Orchestration
Orchestrator application to track the state of a billing process
Kafka as database and Kafka Streams for data processing
Complex stateful aggregations across contracts and re-calculations
Continuous monitoring in real-time
Daily reporting
25
Swiss Streaming Meetup: https://www.youtube.com/watch?v=do-Jl9CHLHA

26. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Humana – Real-Time Integration and Analytics
Interoperability platform to transition from Insurance Company with Elements of Health, to truly a Health Company with Elements of Insurance.
Consumer-centric, health plan agnostic, provider agnostic. Cloud resilient and elastic. Event-driven and real-time.
26
https://www.confluent.io/resources/kafka-summit-2020/levi-bailey-keynote-humana-improving-health-with-event-driven-architectures/

27. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
freeyou – Streaming Analytics
Real-time price adjustments in vehicle insurance
https://www.confluent.io/kafka-summit-san-francisco-2019/how-to-build-real-time-price-adjustments-in-vehicle-insurance-on-streams/

28. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Tesla – Carmaker and Utility Company
Trillions of messages per day for IoT use cases
https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/
https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/

29. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Tesla – Car Insurer
Elon Musk: “much better feedback loop” instead of being statistical like other insurers
Shift from human experts to automation driven
by big data and machine learning
Real-time telematics data from all its drivers’
behavior and the performance of its vehicle
technology (cameras, sensors, …)
Better risk estimation of accidents
and repair costs of vehicles
Evaluation of risk reduction through new
technologies (autopilot, stability control, anti-theft
systems, bullet-resistant steel)

30. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
What is Apache Kafka?

31. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

32. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Kafka Makes Your Business Real-time
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

33. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.

34. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?

35. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Enterprise Data Platform Requirements Are Shifting
1 3 4
2
Scalable for
Transactional Data
Transient Raw data
Built for
Historical Data
Built for Real-
Time Events
Scalable for
ALL data
Persistent +
Durable
Enriched
data
● Value: Trigger real-
time workflows (i.e.
real-time order
management)
● Value: Scale across
the enterprise (i.e.
customer 360)
● Value: Build
mission-critical
apps with zero data
loss (i.e. instant
payments)
● Value: Add context &
situational awareness
(i.e. ride sharing ETA)
35

36. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Only Event Streaming Has All 4 Requirements
Messaging
Databases
Event Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
36
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration

37. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Kafka is Complementary to other Middleware
in the Enterprise Architecture
Orders Customers
Payments
Stock
REST
JMS
ESB
REST
CRM
Mainframe
SOAP
…
Kafka
Kafka
Kafka
Kafka
SOAP
API Management
HTTP
MQ

38. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Project Example: Mainframe Offloading

39. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
“… rescue data off of the mainframe, in a cloud native, microservice-
based fashion … [to] … significantly reduce the reads on the
mainframe, saving RBC fixed infrastructure costs (OPEX). RBC stayed
compliant with bank regulations and business logic, and is now able to
create new applications using the same event-based architecture.”
Mainframe Offloading
for massive cost-savings
https://www.confluent.io/customers/rbc/

40. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Strangler Design Pattern
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/
https://martinfowler.com/bliki/StranglerFigApplication.html
“The most important
reason to consider a
strangler fig
application over a cut-
over rewrite is reduced
risk.”
Martin Fowler

41. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 0: Direct Communication between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)

42. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 1: Kafka for Decoupling between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd
Party CDC Tool

43. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 2 to 4: New Projects and Applications
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project
(Elastic, Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd
Party CDC Tool

44. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 5: Mainframe Replacement
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project
(Elastic, Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
Core Banking ‘2020’
(Modern Technology)
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14

45. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Project Example: Fraud Detection

46. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html

47. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Streaming Analytics for
Claim Processing and Fraud Detection at Scale
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Payment App
Streaming Platform
Analytics Platform
Other Components
Real Time
Alerting
System
All
Data
Alert
Ingest
Data
Human
Intelligence
Analytics
Platform
Train Analytic
Model
Data
Processing
Analytic Model
Preprocess
Data
Consume
Data
Deploy
Analytic Model
www.kai-waehner.de | @KaiWaehner | Streaming Machine Learning in FinServ and Insurance Industries

48. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Ingestion of IoT Data
49
Replication
MirrorMaker /
Confluent Replicator /
Cluster Linking
Kafka Connect
Analytics /
Machine
Learning
Cars
Cars
Cars
Cars
Payment
App
www.kai-waehner.de | @KaiWaehner | Streaming Machine Learning in FinServ and Insurance Industries

49. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
SELECT payment_id, smartphone_id, payment_details
FROM payment p
LEFT JOIN user_database u ON p.smartphone_id = u.smartphone_id
WHERE u.payment_type = 'Apple Pay';
Preprocessing with ksqlDB

50. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed Commit
Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
51
Model X
(at a later time)

51. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Confluent Tiered Storage for Kafka
52
www.kai-waehner.de | @KaiWaehner | Streaming Machine Learning in FinServ and Insurance Industries

52. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Use Cases for Reprocessing Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• New consumer application
• Error-handling
• Compliance / regulatory processing
• Query and analyze existing events
• Schema changes in analytics platform
• Model training
Real-time Consumer
Consumer of Historical Data

53. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
54
“CREATE STREAM FraudDetection AS
SELECT payment_id,
detectAnomaly(payment_values)
FROM payment_table;“

54. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Event Streaming Is The Future Of Data
56
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

55. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Why Confluent?

56. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
58

57. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Car Engine Car Self-driving Car
Confluent Completes Apache Kafka

58. Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
Confluent... Complete. Cloud-native. Everywhere.
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success

59. Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
kai-waehner.de
confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!