Today's modern data architectures and the their implementations contain an Event Broker. What are the benefits of placing an Event Broker in a Modern Data (Analytics) Architecture? What exactly is an Event Broker and what capabilities should it provide? Why is Apache Kafka the most popular realisation of an Event Broker? These and many other questions will be answered in this session. The talk will start with a vendor-neutral definition of the capabilities of an Event Broker. Then the session will highlight the different architecture styles which can be supported using an Event Broker (Kafka), such as Streaming Data Integration, Stream Analytics and Decoupled Event-Driven Applications and how can these be combined into a unified architecture, making the Event Broker the central nervous system of an enterprise architecture. We will end with an overview of the Kafka ecosystem and a placement of the various components onto the Modern Data (Analytics) Architecture.

1. http://guidoschmutz@wordpress.com@gschmutz
Event Broker (Kafka) in Modern Data
Architecture
Guido Schmutz

2. Guido
Working at Trivadis for more than 23 years
Consultant, Trainer, Platform Architect for Java,
Oracle, SOA and Big Data / Fast Data
Oracle Groundbreaker Ambassador & Oracle ACE
Director
@gschmutz guidoschmutz.wordpress.com
210th
edition

4. What exactly is an Event
Broker?

5. Event Broker
Event Broker – as a starting point

6. Event Broker
Event Broker – an Infrastructure with these capabilities
1. topic semantics (publish/subscribe)
– message can be consumed by 0 –
n consumers
2. queue semantics – messages can be
consumed by exactly one consumer
3. horizontally scalable – throughput
increases with more resources
4. auto-scaling – up and down-scaling
upon load
5. highly available – no single point of
failure
6. Control/handle back-pressure
7. durable – messages may not be lost
8. schema-less – no knowledge on
message content and format
9. Efficient support of Stream and
Batch Consumers (offline and with
large Backlog)
10. (Unlimited) Retention of messages
(long term storage)
11. Guaranteed ordering of messages
12. Support re-consumption of events
13. Access control – control over who
can produce and consume which
events
14. interoperable – support for
different clients

7. Kafka – the most popular
Event Broker

8. Kafka – the most popular Event Broker
Kafka Cluster
Consumer 1 Consumer 2
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble
ZK 1 ZK 2ZK 3
Schema
Registry
Service 1
Management
Control Center
Kafka Manager
KAdmin
Producer 1 Producer 2
kafkacat
Data Retention:
• Never
• Time (TTL) or Size-based
• Log-Compacted based
Producer3Producer3
ConsumerConsumer 3
1. topic semantics
2. queue semantics
3. horizontally scalable
4. auto-scaling
5. highly available
6. back-pressure
7. durable
8. schema-less/opaque
9. Stream and Batch Consumers
10. (Unlimited) Retention
11. Guaranteed ordering
12. re-consumption of events
13. Access Control
14. Interoperable

9. • Cloud Services
• Cloud Services with Kafka API
• Kafka Cloud Services
Event Broker - Kafka Alternatives? Cloud Services?
• traditional Message Brokers (with a lot of
limitations regarding Event Broker capabilities)
• Apache Pulsar
• Solace
• Pravega (Dell
Streaming Platform)
• Oracle AQ (Kafka API coming) AQ

10. Event Broker - core building
block of a Modern Data
Architecture

11. Event Broker
Event Broker – as a starting point
Vehicle
Environ
mental
Streaming Data Sources
Ware
house
E-Comm
erce

12. Event Broker
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Streaming Data Sources
Ware
house
Using Stream Data Integration for integrating various
data sources
E-Comm
erce
Stream Data
Integration

13. Event Broker
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Ware
house
Gateway
Using Edge Computing and Stream Data Integration
• MQTT as a gateway to Kafka
E-Comm
erce
Stream Data
Integration
Streaming Data Sources

14. Stream Data Integration – Kafka Connect / StreamSets
• declarative style, simple data flows
• framework is part of Apache Kafka
• Many connectors available
• Single Message Transforms (SMT)
• GUI-based, drag-and drop Data Flow
Pipelines
• Both stream and batch processing (micro-
batching)
• custom sources, sinks, processors

15. Event Broker
Stream
Analytics
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Streaming Data Sources
Ware
house
Using Stream Analytics
• Time Windowed State Management
• Stream-to-Table Joins
• Stream-to-Stream Joins
• Event Pattern Detection
• Machine Learning Model Execution
(Inference)
[1]
E-Comm
erce
Stream Data
Integration
Gateway
19.11 – 13:00 – Kafka Livedemo: Umsetzung einer Streaminglösung #slideless

16. Stream Analytics - Kafka Streams
• Programmatic API, “just” a Java library
• fault-tolerant local state
• Fixed, Sliding and Session Windowing
• Stream-Stream / Stream-Table Joins
• At-least-once and exactly-once
• Stream Processing with zero coding using
SQL-like language
• built on top of Kafka Streams
• interactive (CLI) and headless (cmd file)
trucking_
driver
Kafka Broker
Java Application
Kafka Streams
ksqlDB
trucking_
driver
Kafka Broker
ksqlDB Engine
Kafka Streams
ksqlDB REST
Commands
ksqlDB CLI
push pull

17. Event
Broker
Stream
Analytics
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Ware
house
Using Stream Analytics
• Push results back to new topic so other
interested parties can use it too!
E-Comm
erce
Stream Data
Integration
Streaming Data Sources
Gateway

18. Event
Broker
Stream
Analytics
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Ware
house
Using Stream Data Integration to callback to Data
Source (to Actuator)
E-Comm
erce
Stream Data
Integration
Streaming Data Sources
Gateway

19. Event Broker
Stream
Analytics
Streaming
Visualize
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Ware
house
Using Streaming Visualization
• ksqlDB pull queries or Kafka Streams
Interactive Queries allow to query state of
stream processor
[2]
E-Comm
erce
Stream Data
Integration
Streaming Data Sources
Stream Data
Integration
Gateway

20. Event Broker
Stream
Analytics
Monolithic
System
Stream Data
IntegrationCDC
Streaming
Visualize
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Ware
house
(Right-Time) Legacy Systems Integration
• Stream-to-Table join
E-Comm
erce
Stream Data
Integration
Streaming Data Sources
Gateway
Legacy Data Sources

21. Kafka as an Event Broker

22. Event Broker
Stream
Analytics
Monolithic
System
Machine
IIoT
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Ware
house
(Right-Time) Legacy Systems Integration
E-Comm
erce
Stream Data
Integration
Streaming Data Sources
Gateway
Legacy Data Sources

23. Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Micro-Batch
Visualize
Providing “Materialized Views” in RDBMS or NoSQL
Datastores
Stream Data
Integration
Streaming Data Sources
Gateway
• Bootstrap ”Materialized View” from event history
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

24. Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
1st Micro
service
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Micro-Batch
Visualize
Modern Event-Driven Apps (aka. Microservices)
• Microservice participates as both a
consumer and producer of events
Stream Data
Integration
Streaming Data Sources
Gateway
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

25. Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
1st Micro
service
2nd Micro
service
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Micro-Batch
Visualize
Modern Event-Driven Apps (aka. Microservices)
• 2nd microservice
consumes events
from 1st Bootstrap
from event history
[3]
Stream Data
Integration
Streaming Data Sources
Gateway
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

26. Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
1st Micro
service
2nd Micro
service
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Micro-Batch
Visualize
Bi-Directional Legacy Systems Integration
[4]AQ
Stream Data
Integration
Streaming Data Sources
Gateway
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

27. Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
1st Micro
service
2nd Micro
service
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Micro-Batch
Visualize
Hybrid Cloud, Geo-Distributed or
Disaster Recovery Scenario
AQ
Stream Data
Integration
Streaming Data Sources
Event
Broker
Mirroring
Event Broker
Gateway
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

28. Event Broker
Stream
Analytics
Streaming
Visualize
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
Batch
Analytics
Event Broker as “Virtualized” Data Lake for Batch Analytics
Stream Data
Integration
Streaming Data Sources
1st Micro
service
2nd Micro
service
Stream Data
Integration
NoSQL
RDBMS
Micro-Batch
Visualize
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
AQ
Gateway
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

29. Kafka Storage
Local Storage Tiered Storage (Confluent Enterprise)
Broker 1
Broker 2
Broker 3
Broker 1
Broker 2
Broker 3
Object
Storage
hothot & cold cold
Data Retention:
• Never
• Time (TTL) or Size-based
• Log-Compacted based
1. topic semantics
2. queue semantics
3. horizontally scalable
4. auto-scaling
5. highly available
6. back-pressure
7. durable
8. schema-less/opaque
9. Stream and Batch Consumers
10. (Unlimited) Retention
11. Guaranteed ordering
12. re-consumption of events
13. Access Control
14. Interoperable

30. Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
1st Micro
service
2nd Micro
service
Stream Data
Integration
Stream Data
Integration
Batch Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Data Lake /
DWH
Batch
Visualize
Batch
Analytics
Micro-Batch
Visualize
“Materialized” Data Lake for
Batch Analytics
Stream Data
Integration
Streaming Data Sources
Gateway
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

31. Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
1st Micro
service
2nd Micro
service
Serverless
FaaS
Stream Data
Integration
Stream Data
Integration
Gateway
Batch Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Data Lake /
DWH
Batch
Visualize
Batch
Analytics
Micro-Batch
Visualize
Serverless/Function as a Service (FaaS)
Stream Data
Integration
Streaming Data Sources
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

32. Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
1st Micro
service
2nd Micro
service
Serverless
FaaS
Stream Data
Integration
Stream Data
Integration
Gateway
Batch Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Data Lake /
DWH
Batch
Visualize
Batch
Analytics
Micro-Batch
Visualize
Event Broker becomes the central nervous
system for your information!
Stream Data
Integration
Streaming Data Sources
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

33. Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
Micro
service
Micro
service
Serverless
FaaS
Stream Data
Integration
Stream Data
Integration
Gateway
Batch Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Data Lake /
DWH
Batch
Visualize
Batch
Analytics
Micro-Batch
Visualize
Event Broker becomes the central nervous
system for your information!
Stream Data
Integration
Streaming Data Sources
Log as a first-class citizen!
Turning the database
Inside out!
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce

34. Reference
1. Stream Processing Concepts and Frameworks
2. Streaming Visualization
3. Building event-driven (Micro)Services with Apache Kafka
4. Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka

35. You are welcome to join us at the Expo area.
We're looking forward to meeting you.
Link to the Expo area:
https://www.vinivia-event-
manager.io/e/DOAG/portal/expo/29731
My other talks at DOAG 2020:
18.11 – 10:00 - Big Data, Data Lake, Datenserialisierungsformate
18.11 – 13:00 – Rolle des Event Hubs in einer modernen Daten Architektur
19.11 – 13:00 – Kafka Livedemo: Umsetzung einer Streaminglösung #slideless

36. 36