The Rise of Event Streaming- Why Apache Kafka Changes Everything Kai Waehner, Sr. Systems Engineer, Confluent

1. 1
The Rise Of Event Streaming –
Why Apache Kafka Changes Everything
Kai Waehner | Technology Evangelist, Confluent
contact@kai-waehner.de | LinkedIn | @KaiWaehner | www.confluent.io | www.kai-waehner.de

2. 22
The world is changing.

3. 33
Transformation is
happening everywhere...

4. 4
Transportation & Automotive

5. 5
Banking

6. 6
Largest media co - creates no content
Largest movie house - owns no cinemas
Online bookstore - diversifying across industries
Largest taxi company - owns no vehicles
Most valuable retailer - has no inventory
World’s largest accommodation provider - owns no real estate

7. 77
The New Enterprise Reality
Innovate or be disrupted
● Create new business models
● Deliver new, real-time customer
experiences
● Deliver massive internal
efficiencies
Every company is a
software company
● Capital One
10,000 of 40,000 employees are
software engineers
● Goldman Sachs
1.5B (billion!) lines of code across
7,000+ applications
● JP Morgan: 50+% of IT budget for
new projects and applications,
<50% for maintenance
Innovation is about
events, not “data”
● Digital-first organizations must
pick the right foundational
technologies
● Established enterprises must
implement new technologies
without “rip-and-replace”

8. 8
Every Company is Becoming
a Software Company

9. 9
Every Company is Becoming
a Software Company

10. 1010
What enables this
transformation?

11. 11
The World has Changed
Business Digitalization Trends are Driving the Need to Process
Events at a whole new Scale, Speed and Efficiency

12. 1212
What is the
challenge?

13. 1313
A company is built on
DATA FLOWS
but all we have is
DATA STORES

14. 14
A company built on data stores looks like this

15. 15
ETL/Data Integration Messaging
Highly Scalable
Durable
Persistent
Ordered
Real-time Difficult to Scale
No Persistence After
Consumption
No Replay
Batch
Expensive
Time Consuming

16. 1616
How is it done with an
Event Streaming
Platform

17. 1717
Highly Scalable
Persistent
ETL/Data Integration MessagingETL/Data Integration MessagingMessaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Real-timeHighly Scalable
Durable
Persistent
Ordered
Real-time
Event Streaming

18. 18
Events
A Sale An Invoice A Trade A Customer
Experience

19. 19
An Event Streaming Platform is the
Underpinning of an Event-driven Architecture
Microservices
DBs
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservices
events
SaaS
data
Customer
experience
s
Streams of real time events
Streaming AppsStreaming Apps Streaming Apps

20. 20

21. 21
The beginning of a new Era
https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
The first use case: Log Analytics. This is why Kafka was created!

22. 22
● Global Scale
● Real-time
● Persistent Storage
● Stream Processing
Apache Kafka: The De-facto Standard for
Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache
Kafka

23. 23
Apache Kafka at Scale at Tech Giants
> 7 trillion messages / day > 6 Petabytes / day
“...you name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data

24. 24
Producing to Kafka
P
Time

25. 25
Consuming from Kafka
P
Time
C2
Near
Real Time
C3
Real Time
C1
Batch

26. 26
Distributed System with Replication and High Availability
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

27. 27
Distributed System with Replication and High Availability
read,
write
App
(Kafka Streams)
Kafka
(data)
More Apps
(KSQL, Connect, Python,
REST, “You-name-it”)
BookingsTeam
FraudTeam
…
MobileTeam
…

28. 28
Apache Kafka (kafka.apache.org) includes
Kafka Connect and Kafka Streams
Kafka Streams
Your
app
sinksource
KafkaConnect
KafkaConnect

29. 2929
KSQL
Continuous queries
on infinite streams

30. 3030
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC

31. 31
TRADITIONAL
DATABASE
EVENT STREAM
PROCESSING
SELECT * FROM
DB_TABLE
CREATE TABLE T
AS SELECT * FROM
EVENT_STREAM
Active Query: Passive Data:
DB Table
Active Data: Passive Query:
Event Stream

32. 32
TABLES STREAMS
USER
JAY
SUE
FRED
CREDIT_SCORE
695
430
710V1
V3
V2
PAYMENTS
42
18
65
...
USER
JAY
SUE
FRED
...

33. 33
PUSH PULL
APP
Jay’s credit score is 670
Jay’s credit score is 710
Jay’s credit score is 695
What is Jay’s credit score now?
695
APP

34. 37
The stream is materialized into a table, which is emitted as a stream
Payments Stream
Credit Score Stream
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount)…

35. 3838
What is the result

36. 4040
Event-Driven App
(Location Tracking)
Only Real-Time Events
Messaging Queues can
do this
Contextual
Event-Driven App
(ETA)
Real-Time combined
with stored data
Only Event Streaming
Platforms can do this
Where is my driver? When will my driver
get here?
Where is my driver? When will my driver
get here?
Why add the word
Contextual?

37. 41
Front, rear and top
view cameras
Parking assistant
Environment pointer
Ultrasonic Sensors
Parking assistant with
front and rear camera plus
environment indicator
Crash Sensors
Front protection adaptivity
Side protection
Tail impact protection
Infrared Camera
Rearview assistance with
Pedestrian recognition
Front and Rear
Radar Sensors
ACC with stop and go function
Side assist
Front Camera
Audi Active lane assistant
Speed limit indicator
Adaptive light
The Future of the Automotive Industry
is a Real Time Data Cluster

38. 42
The Future of the Automotive Industry
is a Real Time Data Cluster
Front, rear and top
view cameras
Ultrasonic SensorsCrash Sensors
Front Camera Infrared Camera
Front and Rear
Radar Sensors
Traffic Alerts
Hazard Alerts Personalization
Anomaly
Detection
MQTT MQTT
MQTT
MQTT MQTTMQTT

39. 43

40. 44

41. 454545

42. 464646

43. 474747

44. 484848

45. 494949

46. 52
Real Time Machine Learning with
Kafka and TensorFlow

47. 53
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)

48. 5454
How does Confluent help

49. $$€€££
- Making Money
- Saving Money
- Protecting Money
Every Company
is Becoming
a Software
Company

50. 5656
I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
56
Initial Awareness /
Pilot
Start to Build Pipeline /
Deliver 1 New Outcome
Leverage
Stream Processing
Build Contextual
Event-Driven Apps
Central Nervous
System
Product, Support, Training, Partners, Technical Account Management...

51. 5757
CONFLUENT PLATFORM
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-STAGE
PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
Multi-language Development
Rich Pre-built Ecosystem
SQL-based Stream Processing
GUI-driven Mgmt & Monitoring
Flexible DevOps Automation
Dynamic Performance & Elasticity
Enterprise-grade Security
Data Compatibility
Global Availability
APACHE KAFKA
Fully Managed Cloud ServiceSelf Managed Software
FREEDOM OF CHOICE
COMMITTER-LED EXPERTISE PartnersTraining
Professional
Services
Enterprise
Support
DEVELOPER OPERATOR ARCHITECT
Hybrid Infrastructure

52. 58
Questions?
Let’s connect...
Kai Waehner
Technology Evangelist
kai.waehner@confluent.io
@KaiWaehner
www.confluent.io
www.kai-waehner.de
LinkedIn