Microservices with Kafka Ecosystem

Microservices with Kafka
Ecosystem
Guido Schmutz
@gschmutz doag2017

Guido Schmutz
Working at Trivadis for more than 20 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Microservices with Kafka Ecosystem2

Our company.
Trivadis is a market leader in IT consulting, system integration, solution engineering
and the provision of IT services focusing on and
technologies
in Switzerland, Germany, Austria and Denmark. We offer our services in the following
strategic business fields:
Trivadis Services takes over the interacting operation of your IT systems.
O P E R A T I O N

COPENHAGEN
MUNICH
LAUSANNE
BERN
ZURICH
BRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than
600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:
CHF 5.0 million
Financially self-supporting and
sustainably profitable
Experience from more than 1,900
projects per year at over 800
customers

Agenda
1. Where do we come from?
2. What are Microservices?
3. Why not Event Driven?
4. Why Kafka for Event-Driven Microservices?
5. Asynchronous Microservices in Enterprise Architecture
6. References

Where do we come from?

Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
sync request/response

Shop UI App
Business
Activity Service
SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
Data
GUI
Service
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

Shop UI App
Business
Activity Service
Virtualized SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

What are Microservices?

Key Principles of Microservices
Tightly Scoped
Tightly Encapsulated behind concrete interfaces
Responsible for managing their own data
Highly cohesive
Highly decoupled
Independently deployable, self-contained and autonomous

Microservice Approach Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST

Microservice Approach
with API Gateway
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway

Synchronous World of Request-Response leads to tight,
point-to-point couplings
problem in lower end of chain have a
ripple effect on the other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State

Why not Event-Driven?

3 mechanisms through which services interact
Request-Driven Event Driven
Service
Logic
State
Event
Consume
“IPad OrderedEvent”
Command
”Order IPad”
order(iPad)
Event
Publish
“OrderValidatedEvent”
Query
”Retrieve my Orders)
getAllOrders(myself)
Event Broker

3 mechanisms through which services interact
Commands
• represents an action
• request for some operation to be performed in another service
• Something that will change the state of the system
• Commands expect a response.
Events
• represent both a Fact and a Trigger
• Something that has happened, expressed as a notification
Queries
• represent a request to look something up
• Importantly, queries are side effect free; they leave the state of the system
unchanged

Request-Driven Communication – Highest Coupling
Shop UI Application
UI Logic
Order Microservice
Logic State
{ }
API
Stock Microservice
State
{ }
API Logic
Call PlaceOrderAPI
Call UpdateStockAPI
Maybe Call Reorder API

Decoupling through Events – Lowest Coupling
Event
Broker
Order Microservice
Logic State
{ }
API
Stock Microservice
State
{ }
API Logic3) Raise
OrderConfirmed
Event
4) Listen to
OrderConfirmed
Event
5) Maybe Raise
Reorder Event
2) Listen to
OrderRequested
Event
1) Raise
OrderRequested
Event
async request/response
async, event pub/sub

Event-Driven (Async)
Microservice Approach
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Event
Broker
async request/response
async, event pub/sub

Shop UI App
Business
Activity Service
Process & Virtualized SOA Approach – Sync & Async
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
Orchestration

Command Query Responsibility Segregation (CQRS)
Optimize different nonfunctional requirements
for read and write behavior
interface to the service is split between
• commands that trigger changes in state
• queries that provide read access to the state of
resources
used to support services with higher
performance and capacity requirements for
reading data than for writing data
number of instances supporting queries can
be scaled up independently of instance
supporting commands
Data Storage
Write Data Store
Read Data Store
(Materialized Views)
Service
Command
Service
Query
Service
App
UI
Projection
Service
UI Logic

Event Sourcing
persists the state of a business entity as a
sequence of state-changing events
Whenever state of business entity changes, a
new event is appended to the list of events
Since saving an event is a single operation, it
is inherently atomic
The application reconstructs an entity’s
current state by replaying the events
Data
Storage
Event Store
Service
Event Service
Publisher
App
UI
UI Logic Replayer
Other
App

Event Sourcing & CQRS
Event sourcing is commonly combined
with the CQRS pattern
materializing views from the stored
events
Optionally Commands can be stored in
event store and transformed into events
by the command handler
Data Storage
Event Store
Service
Command Service
App
UI
UI Logic
Query Service Read Data Store
(Materialized Views)
Projection
Service
Command Handler

Using Event Sourcing with Microservices
“Event sourcing enables building
a forward-compatible application
architecture — the ability to add
more applications in the future
that need to process the same
event but create a different
materialized view.”
Neha Narkhede, Confluent Blog
Microservice
State
Command
Handler
{ }
API
REST
Event Store
Projection
Handler(s)
Query Logic
Event Handler(s)
Event Subscribe

How many Event Stores do we need ?
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
OR

Have one source of truth
Avoid double write!
• Would need distributed transactions
Write Event first then consume it from
same micro service
• “eat your own dog food”
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API
StateConsumer
REST
Event Store
Event
Publisher

Data Store
Publish Event through Database
Customer
Event Store
Event
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI

Why Kafka for Event-Driven
Microservices?

Apache Kafka - Unix Analogy
$ cat < in.txt | grep "kafka" | tr a-z A-Z > out.txt
Kafka Connect API Kafka Connect APIKafka Streams API
Kafka Core (Cluster)
Adapted from: Confluent
KSQL

Kafka High Level Architecture
The who is who
• Producers write data to brokers.
• Consumers read data from
brokers.
• All this is distributed.
The data
• Data is stored in topics.
• Topics are split into partitions,
which are replicated.
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble

Leverage the Log
at the heart of Apache Kafka sits a
distributed log
collection of messages, appended
sequentially to a file
service ‘seeks’ to the position of the last
message it read, then scans sequentially,
reading messages in order
log-structured character makes Kafka
well suited to performing the role of an
Event Store in Event Sourcing
Event Hub
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22
Reads are a single
seek & scan
Writes are
append only

Scale-Out Architecture
topic consists of
many partitions
producer load
load-balanced
over all partitions
consumer can
consume with as
many threads as
there are
partitions
Producer 1
Consumer 1
Broker 1
Producer 2
Producer 3
Broker 2
Broker 3
Consumer 2
Consumer 3
Consumer 4
Consumer Group 1
Consumer Group 2
Kafka Cluster

Strong Ordering Guarantees
most business systems need strong
ordering guarantees
messages that require relative
ordering need to be sent to the same
partition
supply same key for
all messages that
require a relative order
To maintain global ordering use a
single partition topic
Producer 1
Consumer 1
Broker 1
Broker 2
Broker 3
Consumer 2
Consumer 3
Key-1
Key-2
Key-3
Key-4
Key-5
Key-6
Key-3
Key-1

Durable and Highly Available Messaging
Producer 1
Broker 1
Broker 2
Broker 3
Producer 1
Broker 1
Broker 2
Broker 3
Consumer 1 Consumer 1
Consumer 2Consumer 2

Durable and Highly Available Messaging (II)
Producer 1
Broker 1
Broker 2
Broker 3
Producer 1
Broker 1
Broker 2
Broker 3
Consumer 1 Consumer 1
Consumer
2
Consumer 2

Replay-ability – Logs never forget
by keeping events in a log, we have a
version control system for our data
if you were to deploy a faulty program,
the system might become corrupted, but
it would always be recoverable
sequence of events provides an audit
point, so that you can examine exactly
what happened
rewind and reply events, once service is
back and bug is fixed
Event Hub
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22
Replay
Rewind
Service
Logic State

Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL)
log.retention.{ms | minutes | hours}
3. Size based
log.retention.bytes
4. Log compaction based
(entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181
--create --topic customers
--replication-factor 1
--partitions 1
--config cleanup.policy=compact

Keep Topics in Compacted Form
0 1 2 3 4 5 6 7 8 9 10 11
K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11
Offset
Key
Value
3 4 6 8 9 10
K1 K3 K4 K5 K2 K6
V4 V5 V7 V9 V10 V11
Offset
Key
Value
Compaction
V1 V2
V3
V4 V5
V6
V7
V8
V9V10 V11
K1 K3 K4 K5K2 K6

Topic Viewed as Event Stream or State Stream (Change
Log)
Event Stream State Stream (Change Log Stream)
2017-10-02T20:18:46 11,Normal,41.87,-87.67
2017-10-02T20:18:55 11,Normal,40.38,-89.17
2017-10-02T20:18:59 21,Normal,42.23,-91.78
2017-10-02T20:19:01 21,Normal,41.71,-91.32
2017-10-02T20:19:02 11,Normal,38.65,-90.2
2017-10-02T20:19:23 21,Normal41.71,-91.32
11 2017-10-02T20:18:46,11,Normal,41.87,-87.67
11 2017-10-02T20:18:55,11,Normal,40.38,-89.17
21 2017-10-02T20:18:59, 21,Normal,42.23,-91.78
21 2017-10-02T20:19:01,21,Normal,41.71,-91.32
11 2017-10-02T20:19:02,11,Normal,38.65,-90.2
21 2017-10-02T20:19:23,21,Normal41.71,-91.32

Keep Topics both in Original and Compacted Form
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Consume and Produce
OR
TX

Legacy Microservice
Change Data Capture (CDC)
RDBMS cdc-source trucking_
driver
Driver Topic
elasticsearch
-sink NoSQL

Enrich Stream with Static Data with Kafka Streams
Movement Topic Consume and Produce
Driver
Table
Driver Topic
Driver
Handler
Join
Driver Local State
Movement & Driver Topic

Building Embedded, Materialized View with Kafka
Streams
Consume and Produce
Join
Store
Engine Metric Topic
Join
Join State
Window
Aggregation
Result
Store
Result State

Building Embedded, Queryable Materialized View with
Kafka Streams
Consume and Produce
Join
Store
Engine Metric Topic
Join
Join State
Window
Aggregation
Result
Store
Result State
API Application

Asynchronous Microservices in
Enterprise Architecture

Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Asynchronous Microservices in Enterprise Architecture
Location
Social
Click
stream
Sensor
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Call
Center
Mobile
Apps
SQL
Search
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Weather
Data
Event
Hub
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice Cluster
Microservice State
{ }
API
Stream Analytics Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Service

References

References
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:
• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc

Technology on its own won't help you.
You need to know how to use it properly.

Trivadis @ DOAG 2017
#opencompany
Booth: 3rd Floor – next to the escalator
We share our Know how!
Just come across, Live-Presentations
and documents archive
T-Shirts, Contest and much more
We look forward to your visit

Microservices with Kafka Ecosystem

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Microservices with Kafka Ecosystem

Similar to Microservices with Kafka Ecosystem (20)

More from Guido Schmutz

More from Guido Schmutz (20)

Recently uploaded

Recently uploaded (20)

Microservices with Kafka Ecosystem