What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

1. BASEL | BERN | BRUGG | BUCHAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENEVA
HAMBURG | COPENHAGEN | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH
http://guidoschmutz.wordpress.com@gschmutz
Building event-driven (Micro)Services with
Apache Kafka
Guido Schmutz
OiO-Hauskonferenz – 19.12.2019

2. Agenda
1. Where do we come from?
2. What about Microservices?
3. Can we do better?
4. How does Apache Kafka help?
5. What about Streaming Data Sources?
6. What about integrating Legacy Applications?
7. What about (historical) data analytics?
8. Summary

3. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF
HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH
Guido
Working at Trivadis for more than 23 years
Consultant, Trainer, Platform Architect for Java,
Oracle, SOA and Big Data / Fast Data
Oracle Groundbreaker Ambassador & Oracle ACE
Director
@gschmutz guidoschmutz.wordpress.com
178th
edition

4. Where do we come from?

5. Shop Rich UI
Shop Backend Application
“Layered Architecture” Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
sync request/response

6. Shop UI App
Business
Activity Service
SOA Approach
• Contract-
first Web
Services
• Technical
layers offer
their own
interfaces
• Reuse on
each level
• Lower layer
often
wraps
legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
GUI
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

7. Business
Activity Service
Virtualized SOA Approach
• The raise of the
Enterprise
Service Bus (ESB)
Search BAS
Customer DAO
Order DAO
Order BAS
Business Entity
Service
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
Shop UI App
Shop UI UI Logic
GUI
Shop Web App
Shop UI UI Logic
GUI
X

8. Shop UI App
Business
Activity Service
Orchestrated & Virtualized SOA Approach – Sync/Async
• The raise of
orchestration
engines
• (BPEL & BPMN)
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
X
Orchestration

9. What about Microservices?

10. Customer Microservice
Microservice Approach
• Tightly Scoped behind interfaces
• Highly decoupled
• Independently deployable
• Bounded Context/Aggregate
(DDD)
• Responsible for their data (does
not mean they need their own
DB!)
• Smart Endpoints and Dump
Pipes
• just SOA done right ? What
about capabilities the “smart
pipes” provided?
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST

11. Service 1
Service 2
Service 3
Service 4
Smart Endpoints and Dumb Pipes! - no longer a
complex Integration in the middle!
Integration
X
Service Bus
Service
Orchestration
Service 1
Service 2
Service 3
Service 4
X
X
X
X

12. Synchronous Request-Response lead to tight, point-
to-point couplings
problem in lower end of chain have a ripple
effect on other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State
RESTRESTRESTREST
REST REST REST

13. Microservice Approach with
API Gateway
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
X

14. Microservice Approach with Side Car (i.e. K8s & Istio)
like AOP but on infrastructure
can be framework or proxy (side car) based
Service Mesh provides:
• retry, load-balancing, circuit breaker,
throttling, security, tracing, …
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Side
Car
Side
Car
Side
Car
REST
RESTRESTREST

15. Microservice Approach with Side Car (i.e. K8s & Istio)
• Side-car and it’s advanced routing
capabilities can be used to switch to new
service version
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3 – v1
{ }
API Logic
State
Side
Car
Side
Car
Service 3 – v2
{ }
API Logic
State
RESTREST REST
REST

16. Side Car provides lot of value …. but
• we still have to change the “data owner”
service ….
if a new service requires the same
information
Service 1
{ }
API Logic
State
Service 2
{ }
API Logic
State
Service 4
Logic
State
Service 3
{ }
API Logic
State
{ }
API
New Service
Logic
State
{ }
API
REST
REST REST REST REST

17. Can we do even(t) better?

18. Events
Distribute to all handlers
strong ordering req’s
No results
Queries
Route with load balancing
Sometimes scatter-gather
Provide result
Three mechanisms through which services can
interact
Commands
Route to single handler
Use consistent hashing
Provide Result
Adapted from Axon IQ

19. Stock Microservice
Event-Driven (Async) Microservice
Event Hub
• Pub / Sub messaging
• Message transparent to topic
• Message coupling
• Domain Events from DDD
• This is event-driven interaction but not
event sourcing!
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
REST
REST
REST
REST
API
Gateway
Event
Hub
sync request/response
async request/response
async, event pub/sub
Customer
Mat View

20. Customer Microservice
Stock Microservice
Event-Driven (Async) Microservice
Consumer Microservice is
responsible for adapting
to subscribed event, if
necessary
does not have to use
same technologies
Anti Corruption Layer
(ACL)
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
{ }
Stock API
Stock
Stock Logic
REST
REST
REST
REST
Customer
Mat View
Event
Hub
Consume
X
Consume
X
Consume
X
Consume
X

21. How does Apache Kafka help?

22. Apache Kafka – highly scalable message broker
Kafka Cluster
Consumer 1 Consume 2r
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble
ZK 1 ZK 2ZK 3
Schema
Registry
Service 1
Management
Control Center
Kafka Manager
KAdmin
Producer 1 Producer 2
kafkacat
Data Retention:
• Never
• Time (TTL) or Size-based
• Log-Compacted based
Producer3Producer3
ConsumerConsumer 3
Event Hub

23. Example
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Microservice
{ }
Order API OrderOrder Logic
REST REST
Event
Hub
Customer
Mat View
Schema
Registry
Schema
Customer
(compacted)
Implementation: https://github.com/gschmutz/event-driven-microservices-demo
in functional terms this is an implementation of CQRS pattern => System-Wide CQRS

24. Apache Avro
• http://avro.apache.org/docs/current/
• Apache Avro™ is a compact, fast,
binary data serialization system
invented by the makers of Hadoop
• Avro relies on schemas.
• When data is read, the schema used
when writing needs to be present
• container file for storing persistent
data
• Works both with code generation and
in a dynamic manner
{
"type" : "record",
"namespace" : "com.trivadis.avro.customer.v1",
"name" : ”Customer",
"description" : "the representation of a customr",
"fields" : [
{ "name": "id", "type": ”string" },
{ "name": "firstName", "type": "string" },
{ "name": "lastName", "type": "string" },
{ "name" : "title", "type" : {
"type" : "enum", "name" : "TitleEnum",
"symbols" : ["Unknown", "Mr", "Mrs", "Ms"]
}
},
{ "name": "email", "type": ["null","string"] },
{ "name": "dateOfBirth", "type": {
"type": "int", "logicalType": "date" } },
{ "name" : "addresses", ... }
]
}

25. Avro and Confluent Schema Registry

26. Example
@Configuration
public class KafkaConfig {
private String bootstrapServers;
private String schemaRegistryURL;
@Bean
public Map<String, Object> producerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryURL);
return props;
}
@Bean
public ProducerFactory<String, Customer> producerFactory() { .. }
@Bean
public KafkaTemplate<String, Customer> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
@Component
public class CustomerEventProducer {
@Autowired
private KafkaTemplate<String, Person> kafkaTemplate;
@Value("${kafka.topic.customer}")
String kafkaTopic;
public void produce(Customer customer) {
kafkaTemplate.send(kafkaTopic, customer.getId().toString(), customer);
}
}

27. Adding a new service by
bootstrapping from Event Hub
Customer Search
Microservice
{ }
Customer API CustomerCustomer Logic
REST
Schema
Registry
Schema
Consume
from Offset 0
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Microservice
{ }
Order API OrderOrder Logic
REST REST
Event
Hub
Customer
Mat View
Customer
(compacted)
Implementation: https://github.com/gschmutz/event-driven-microservices-demo
• assuming Event Hub keeps message
log as long as needed, new services
can be bootstrapped with
information from other services just
out of the Event Hub

28. Log Compaction – specially useful for Master Data
0 1 2 3 4 5 6 7 8 9 10 11
11 12 11 14 13 12 11 15 15 12 16 12
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12
Offset
Key
Value
3 4 6 8 9 10
14 13 11 15 12 16
V4 V5 V7 V9 V10 V11
Offset
Key
Value
Compaction
V1
V2
V3 V7
V5
V6
V4
V8 V9
V10
V11
11
13
14
15
12
16
Id: 11
Name: Peter Muster
Street: Münstergasse 5
City: 3013 Bern
Category: B
Id: 11
Name: Peter Muster
Street: Chaumonweg 108
City: 3098 Spiegel
Category: B
Id: 11
Name: Peter Muster
Street: Chaumonweg 108
City: 3098 Spiegel
Category: A
Id: 11
Name: Peter Muster
Street: Chaumonweg 108
City: 3098 Spiegel
Category: A
11
12
V12

29. Event-Driven (Async) Microservice
• Event Hub and Service
Mesh
• not yet feasible, as Event
Hub protocols are not
supported by todays
service mesh
• could be used for
versioning, security,
tracing, monitoring …
sync request/response
async request/response
async, event pub/sub
Side Car
Side Car
Side Car
Stock Microservice
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
{ }
Stock API
Stock
Stock Logic
REST
REST
REST
REST
Customer
Mat View
Side Car
Event
Hub

30. What about Streaming Data
Sources?

31. How to work with
streaming data sources
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Event
Hub
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Event Stream

32. Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Streaming Processing & Microservices Architecture
BI Tools
SQ
L
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Analytics
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Hub
Service

33. Apache Kafka – scalable message processing
and more!
Source
Connector
Kafka Broker
Sink
Connector
Stream
Processing
Schema
Registry
Kafka Kafka
ksqlDB

34. Kafka Connect
Source
Connector
Sink
Connector

35. Kafka Streams
• Programmatic API, “just” a Java library
• Native streaming
• fault-tolerant local state
• Fixed, Sliding and Session Windowing
• Stream-Stream / Stream-Table Joins
• At-least-once and exactly-once
KTable<Integer, Customer> customers = builder.stream(”customer");
KStream<Integer, Order> orders = builder.stream(”order");
KStream<Integer, String> joined = orders.leftJoin(customers, …);
joined.to(”orderEnriched");
trucking_
driver
Kafka Broker
Java Application
Kafka Streams

36. ksqlDB
• Stream Processing with zero coding using SQL-
like language (now supporting push and pull
queries)
• part of Confluent Platform (community
edition)
• built on top of Kafka Streams
• interactive (CLI) and headless (command file)
CREATE STREAM customer_s WITH (kafka_topic='customer', value_format='AVRO');
SELECT * FROM customer_s WHERE address->country = 'Switzerland';
...
trucking_
driver
Kafka Broker
ksqlDB Engine
Kafka Streams
ksqlDB REST
Commands
ksqlDB CLI
push pull
ksqlDB

37. What about integrating
Legacy Applications?

38. Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Integrate existing systems through CDC
BI Tools
SQ
L
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Processor
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Hub
Service
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture

39. Data Store
Integrate existing systems through CDC
• Capture changes directly on database
• Change Data Capture (CDC) => think like a
database-wide trigger
• Make existing legacy systems an event
producer
Customer
Event Hub
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI
Stream Processing
Results
Stream Processor
Reference /
Models
Dashboard

40. Legacy Microservice
Change Data Capture (CDC) with
Kafka Broker and Kafka Connect
RDBMS cdc-source trucking_
driver
Kafka Broker
elasticsearch-
sink
NoSQL
Customer Topic

41. Microservice
Change Data Capture (CDC) with
Kafka Broker and Kafka Connect
cdc-source
elasticsearch-
sink
NoSQL
Enhance
Kafka Broker
Customer Topic
CustomerEnhanced Topic
State
Legacy
RDBMS

42. What about (historical) data
analytics?

43. Streaming & (Big) Data Analytics Architecture
Event
Stream
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
D
ata
Flow
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice Cluster
Microservice State
{ }
API
Stream Processing Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
SQL
Export
SearchEvent
Hub
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture
File Import / SQL Import

44. Summary

45. Hadoop Clusterd
Hadoop Cluster
Big Data
Summary
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Event
Hub
D
ata
Flow
D
ata
Flow
Change
Data
Capture Parallel
Processing
Storage
Storage
RawRefined
Results
SQL
Export
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Search
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Stream Processing
Microservices

46. Summary
• not all communication need to be synchronous => distinguish into commands, events, queries
• Events should use a schema with support for backward and forward compatibility
• Kafka handles event streaming very well
• brings many more interesting features beyond just “message passing”, i.e. Log
compaction
• Kafka broker is not a full-fledged Event Store
• For Event Sourcing additional capabilities are needed (Kafka Streams, Axon, …)
• See also: Kafka as an Event Store: is it good enough? (slides, video)

47. Further Information
• Kafka as an Event Store: is it good enough?, Guido Schmutz, Trivadis:
https://www.slideshare.net/gschmutz/kafka-as-an-event-store-is-it-good-enough
• Microservices Blog Series, Ben Stopford, Confluent:
https://www.confluent.io/blog/tag/microservices
• Apache Kafka for Microservices: A Confluent Online Talk Series: https://www.confluent.io/landing-
page/microservices-online-talk-series/
• Schemas, Contracts, and Compatibility, Gwen Shapira, Confluent:
https://www.confluent.io/blog/schemas-contracts-compatibility
• Should You Put Several Event Types in the Same Kafka Topic?, Martin Kleppmann:
https://www.confluent.io/blog/put-several-event-types-kafka-topic/
• Turning the database inside-out with Apache Samza, Martin Kleppmann:
https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
• Immutability Changes Everything, Pat Helland, Salesforce:
http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf