Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement?
This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
Optimizing AI for immediate response in Smart CCTV
ย
Kafka as an event store - is it good enough?
1. BASEL | BERN | BRUGG | BUKAREST | DรSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF
HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MรNCHEN | STUTTGART | WIEN | ZรRICH
http://guidoschmutz.wordpress.comgschmutz
Kafka as an Event Store โ is it Good Enough?
Guido Schmutz
W-JAX Munich โ 6.11.2019
2. gschmutz
Agenda
1. How do we build applications traditionally?
2. CQRS & Event Sourcing
3. What exactly is an Event Store?
4. Implementing Event Store
5. Summary
3. BASEL | BERN | BRUGG | BUKAREST | DรSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF
HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MรNCHEN | STUTTGART | WIEN | ZรRICH
Guido
Working at Trivadis for more than 22 years
Consultant, Trainer, Platform Architect for Java,
Oracle, SOA and Big Data / Fast Data
Oracle Groundbreaker Ambassador & Oracle ACE
Director
@gschmutz guidoschmutz.wordpress.com
173rd
edition
5. gschmutz
Data Access Layer
Monolithic System - using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
6. gschmutz
Data Access Layer
Monolithic System - using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
Object/Relational
Impedance Mismatch
โข Traditional approach
to persistence
โข Store current state
โข CRUD operations
โข Coupling between
read & write
โข Increased Complexity
7. gschmutz
Data Access Layer
Monolithic System - using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
โข Traditional approach
to persistence
โข Store current state
โข CRUD operations
โข Coupling between
read & write
โข Increased Complexity
SELECT acc.id, acc.account_number, acc.account_type.id,
acct.name, cus.first_name, cus.last_name,
addr.street, addr.city
FROM account_t acc
, account_type_t acct
, customer_t cus
, cust_adr_t cusa
, address_t addr
WHERE acc.account_type_id = acct.account_type_id
AND adc.customer_id = cus.customer_id
AND cusa.customer_id = cus.customer_id
AND cusa.address_id = addr.address_id
AND cusa.type = 'MAIN'
8. gschmutz
Microservices - using Layered Architecture
Microservices โฆ
โข are responsible for their data
โข might use NoSQL instead of
RDBMS
โข often still use traditional
approach to persistence
โข โData silosโ do no longer support
database join
โข keep synchronous
communication to a minimum
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Account Microservice
{ }
Account API
Account
Account Logic
Product Microservice
{ }
Product API
Product
Product Logic
Finance App
Finance UI UI Logic
GUI
REST
REST
REST
sync request/response
async, event pub/sub
9. gschmutz
Events
Distribute to all handlers
strong ordering reqโs
No results
Queries
Route with load balancing
Sometimes scatter-gather
Provide result
Three mechanisms through which services can
interact
Commands
Route to single handler
Use consistent hashing
Provide Result
Adapted from Axon IQ
10. gschmutz
Domain Driven Design (DDD) โ Concepts
โข Domain Objects โ hold the state of the
application
โข Entity โ Domain Objects with an identity
โข Value Object โ an immutable type that is
distinguishable only by the state of its
properties and has no identity
โข Aggregate - A cluster of domain objects
that can be treated as a single unit
โข Aggregate Root โ one object of aggregate
is root object. Any reference from outside
goes through aggregate root
Aggregate Root
Account Aggregate
Customer
Aggregate
Aggregate Root
11. gschmutz
Microservices with Event-driven communication
This is Event Streaming and
not really Event Sourcing
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Account Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
REST
REST
REST
Event Hub
(pub/sub)
Customer
Mat View
sync request/response
async, event pub/sub
Finance App
Finance UI UI Logic
GUI
13. gschmutz
Command Query Responsibility Segregation
(CQRS)
Optimize for write and read differently
API is split between
โข commands - trigger changes in state
โข queries - provide read access to the state
Still using CRUD pattern, but separates
โRโ from CRUD
Might involve eventual consistency
between write and read model
Data Storage
Write Model
Read Model
(read-only)
Service
Command
API
Query
API
App
UI
Projection
Handler
UI Logic
CDC
command
query
project
read
insert
update
delete
1
2
3
4
14. gschmutz
Local CQRS vs. System Wide CQRS
Local CQRS System-Wide CQRS
Write Model
Read Model
(read-only)
Service
Command
API
Query
API
App
UI
Projection
Handler
UI Logic
CDC
command
query
project
read
insert
update
delete
Write Model
Read Model
(read-only)
Service
Command
API
App
UI
Projection
Handler
UI Logic
CDC
command
project
insert
update
delete
Service
Write Model
command
Command
API
insert
update
delete
16. gschmutz
Event Sourcing
persists the state of an aggregate as a
sequence of state-changing events
Each event describes a state change
that occurred to the aggregate in the
past
new event is appended to the list of
events
an aggregateโs current state is
reconstructed by replaying the events
=> a.k.a โrehydrationโ
Rehydration also needed for queries
Event Store
ServiceApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
Service
Subscribe
publish
publish
apply (append)
REST
Data Storage
trigger replycommand
command
1 2
3
4
5
5
17. gschmutz
Event Sourcing - โRehydrateโ State
1. Create an empty Aggregate
object
2. Read all events stored for
that Aggregate from event
store
3. Apply each event to the
Aggregate object in the
correct order
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
Account
<empty>
Account
id: 123
accountType: Savings
balance: 0
Account
id: 123
accountType: Savings
balance: 3000
transactions: [+1000, +2000]
Account
id: 123
accountType: Savings
balance: 2500
transactions:[+1000, +2000, -500]
Account
id: 123
accountType: Savings
balance: 1000
transactions: [+1000]
applyTo
applyTo
applyTo
applyTo
18. gschmutz
Event Sourcing โ Write Path CreateAccount
command
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
AccountCreated
id: 123
accountType: Savings
10
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
Account Aggregate
<empty>
19. gschmutz
Event Sourcing โ Write Path DepositMoney
command
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
10 20
Account Aggregate
id: 123
accountType: Savings
balance: 0
20. gschmutz
Event Sourcing โ Write Path DepositMoney
command
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
10 20 100
Account Aggregate
id: 123
accountType: Savings
balance: 1000
22. gschmutz
Event Sourcing - Potential Benefits
1. Subscribe to changes from other
Aggregates
2. Examine a historical record of every
change that has ever been applied
on the model
3. Use the event store data for trend,
forcast and other business analytics
4. Consider โwhat ifโ questions by
replaying events to Aggregates which
have experimental enhancements
5. Patch errors by adding โcorrectionโ
events (if it is legally allowed)
6. Perform โundoโ and โredoโ
operations by replying varying sets
of Events
23. gschmutz
Event Sourcing & CQRS
Event sourcing is commonly
combined with the CQRS pattern
Combines best of Event Sourcing and
CQRS
Project events published by Event
Store into Read Model (Materialized
Views)
Write Model and Read Model might
only support eventual consistency
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection
Handler
command
query read
1
Event Store
publish
apply (append)
trigger reply
2
3
4
5
publish
project
5
6
26. gschmutz
Event Store Capabilities
1. Append Events efficiently
2. Read aggregateโs events in order
3. Full Sequential Read (over all
aggregates)
4. Consistent writes
5. Event versioning
6. Subscribable event stream
7. Correction events (O)
8. Ingestion & event time, bi-
temporal (O)
9. Adhoc-Query on event store (O)
10.Snapshot Optimization (O)
11.High-Availability and Reliability (O)
27. gschmutz
How many Event Stores do we need ?
{ }
API
State
Logic
REST
Event Store
{ }
API
State
Logic
REST
Event Store
Microservice
{ }
API
State
Logic
REST
Event Store
Microservice
{ }
API
State
Logic
REST
Microservice
{ }
API
State
Logic
REST
Event Store
Event
Microservice
{ }
API
State
Logic
REST
OR
Microservice
Microservice
29. gschmutz
Event Store Implementations
โข Event Store (https://eventstore.org/) โ by Greg Young
โข Axon Framework & Relational DB (https://axoniq.io/) - by Axon IQ
โข Axon DB (https://axoniq.io/) - by Axon IQ
โข Eventuate (https://eventuate.io/) โ by Eventuate.io
โข Serialized (https://serialized.io/) โ by Serialized.io
โข Build your own โฆ.
โข Apache Kafka ???
31. gschmutz
Apache Kafka โ A Streaming Platform
Kafka Cluster
Consumer 1 Consume 2r
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble
ZK 1 ZK 2ZK 3
Schema
Registry
Service 1
Management
Control Center
Kafka Manager
KAdmin
Producer 1 Producer 2
kafkacat
Data Retention:
โข Never
โข Time (TTL) or Size-based
โข Log-Compacted based
Producer3Producer3
ConsumerConsumer 3
32. gschmutz
No SPoF, highly available
Consumer polls for new messages based on
offset
Apache Kafka โ A Streaming Platform
horizontally scalable, guaranteed order
33. gschmutz
Kafka as an Event Store
1. One, single-partitioned Kafka topic per Aggregate
2. One, partitioned Kafka topic per Aggregate Type
3. One single, highly partitioned Kafka topic for all Aggregate Types
Should you put several Event Types in the same Kafka topic?:
https://www.confluent.io/blog/put-several-event-types-kafka-topic/
34. gschmutz
1) One, single-partitioned Kafka topic per
Aggregate Instance
This will guarantee that the events are stored
in order
Reading state of an aggregate is as simple as
reading a topic from offset 0
Not really feasible as there will be just too
many topics needed
Kafka
Customer Aggregate
Account Aggregate
35. gschmutz
2) One, partitioned Kafka topic per Aggregate
Type
โข Required number of partitions is dependent
on number of aggregate instances
โข Events are produced with aggregate-id as the
key
โข guarantees that events are stored in order
โข For reading state of an aggregate, all data of
all aggregate instances have to be scanned =>
slow
โข Possible optimization: only read the partition
where aggregate instance is stored
Kafka
Customer Aggregate
Account Aggregate
36. gschmutz
3) One single, highly partitioned Kafka topic for
all Aggregate Types
โข Required number of partitions is dependent
on number of aggregate types * instances
โข Events are produced with aggregate-id as the
key
โข guarantees that events are stored in order
โข For reading state of an aggregate, all data of
all aggregate types & instances have to be
scanned => really slow
โข Possible optimization: only read the partition
where aggregate instance is stored
Kafka
Customer Aggregate
Account Aggregate
37. gschmutz
Kafka as an Event Store
# Capability Kafka Broker
1 Append events efficiently yes
2 Read aggregateโs events in order not efficiently
3 Full sequential Read yes
4 Consistent Writes no
5 Event Versioning yes (if Avro is used)
6 Subscribeable Event Stream yes
7 Correction events (O) no
8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header
9 Snapshot Optimization (O) no
10 Ad-Hoc Query on Events (O) no
11 High-Availability and Reliability (O) yes
38. gschmutz
Event Store
Kafka is not a Database โฆ a Database is not
Kafka
We can use Kafka to run part of our own Event
Store implementation
add a database to get missing capabilities
But be careful with Dual Write!
โข Would need distributed transactions
โข Otherwise no guarantee for both writes to
happen
Application
{ }
API DatabaseBiz Logic
REST
Event Hub
Other App
Consumer
39. gschmutz
Event Store
Kafka is not a Database โฆ a Database is not
Kafka
We can use Kafka to run our own Event Store
implementation
adding a database to get missing capabilities
But be careful with Dual Write!
โข Would need distributed transactions
โข Otherwise no guarantee for both writes to
happen
Application
{ }
API DatabaseBiz Logic
REST
Event Hub
Other App
Consumer
40. gschmutz
Event StoreEvent Store
Two solutions for avoiding ยซdual writeยป
Write Event first then consume it to write it to
database
Write through database (CDC, outbox design
pattern)
Application
{ }
API
Database
Biz Logic
REST
Event Hub
Other App
Biz Logic
Application
{ }
API
Database
REST
Biz Logic
CDC
Event Hub
CDC
Connector
Other App
Biz Logic
Publish
42. gschmutz
Axon
โข Spring Boot with Axon Framework
for Application
โข MongoDB for Event Store
โข Kafka Broker for Event Bus
โข Kafka Streams or KSQL for Projection
Handler
โข Kafka Connect / Spring Boot to
persist in read model
โข NoSQL and/or RDBMS for read
model
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply
44. gschmutz
Event Sourcing with Axon - Aggregate
@Aggregate
public class AccountAggregate{
@AggregateIdentifier
private String id;
private BigDecimal balance;
private String forCustomerId;
private String accountType;
@CommandHandler
...
@EventSourcingHandler
...
45. gschmutz
Event Sourcing with Axon - Command Handler
@CommandHandler
public AccountAggregate(AccountCreateCommand command)
{
Assert.hasLength(command.getForCustomerId(),
"CustomerId must have a value");
Assert.hasLength(command.getAccountType(),
"AccountType must have a value");
...
apply(new AccountCreatedEvent(command.getId(),
command.getForCustomerId(),
command.getAccountType(),
new BigDecimal("0")));
}
46. gschmutz
Event Sourcing with Axon โ Command Handler
@CommandHandler
public void on(WithdrawMoneyCommand command) {
Assert.isTrue(command.getAmount() > 0,
"Amount should be a positive number");
if(command.getAmount().compareTo(this.balance) > 0 ) {
throw new InsufficientBalanceException(
"Insufficient balance. Trying to withdraw:" +
command.getAmount() +
", but current balance is: " + this.balance);
}
apply(new MoneyWithdrawnEvent(command.getId(),
command.getAmount()));
}
47. gschmutz
Event Sourcing with Axon โ Event Handler
@EventSourcingHandler
public void handle(AccountCreatedEvent event) {
id = event.getId();
forCustomerId = event.getForCustomerId();
accountType = event.getAccountType();
balance = event.getBalance();
}
@EventSourcingHandler
public void handle(MoneyWithdrawnEvent event) {
balance = balance.subtract(event.getAmount());
}
48. gschmutz
Event Sourcing with Axon โ Projection Handler
public class AccountQueryController {
@Autowired
private AccountRepository accRepo;
@EventHandler
public void on(AccountCreatedEvent event,@Timestamp Instant instant) {
Account account = new Account(event.getId(),event.getBalance(),
event.getAccHolder(),event.getAccHolderName(),
instant.toString());
accRepo.insert(account);
}
@EventHandler
public void on(MoneyDepositedEvent event,@Timestamp Instant instant) {
Account account = accRepo.findByAccountNo(event.getId());
account.setBalance(account.getBalance().add(event.getAmount()));
account.setLastUpdated(instant.toString());
accRepo.save(account);
}
49. gschmutz
Axon with Axon DB
โข Spring Boot with Axon Framework
for Application
โข Axon DB for Event Store and Event
Bus
โข Spring Boot for Projection Handler
โข Spring Boot to persist in read
model
โข NoSQL and/or RDBMS for read
model
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply
50. gschmutz
Axon as an Event Store
# Capability Axon Framework Axon Framework & Axon DB
1 Append events efficiently yes yes
2 Read aggregateโs events in order yes yes
3 Full sequential Read yes yes
4 Consistent Writes yes yes
5 Event Versioning yes yes
6 Subscribeable Event Stream yes yes
7 Correction events (O) no no
8 Event time & ingestion time, aka. Bi-temporal (O) no no
9 Snapshot Optimization (O) yes yes
10 Ad-Hoc Query on Events (O) yes yes
11 High-Availability and Reliability (O) possible yes
53. gschmutz
Kafka & Kafka Streams
โข Kafka Streams with State for Event
Store
โข Kafka Broker for Event Bus
โข Kafka Streams or KSQL for
Projection Handler
โข No reply of events, current
snapshot is held in state store
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply
55. gschmutz
Kafka & Kafka Streams as an Event Store
# Capability Kafka & Kafka Streams
1 Append events efficiently yes
2 Read aggregateโs events in order no (snapshot state only holds current snapshot)
3 Full sequential Read no
4 Consistent Writes yes (only one event per aggregate in flight)
5 Event Versioning yes (if Avro used)
6 Subscribeable Event Stream yes
7 Correction events (O) no
8 Event time & ingestion time, aka. Bi-temporal (O) no
9 Snapshot Optimization (O) yes (snapshot state only)
10 Ad-Hoc Query on events (O) limited (KSQL, Presto on Kafka, Drill on Kafka, โฆ)
11 High-Availability and Reliability (O) yes
57. gschmutz
Summary
โข Event Sourcing and CQRS might be more natural to business people than
IT => we are used to work with โCRUD based persistenceโ
โข Event Sourcing provides history and logging for free
โข Kafka Broker alone is really โjustโ Event Streaming, not Event Sourcing
โข Axon Framework supports the implementation of Event Sourcing
applications with Pluggable Event Store and Event Bus implementations
โข Axon DB implements an Event Store and an Event Bus
โข Kafka and Kafka Streams with State Store supports event sourcing in a
โstreaming fashionโ with current snapshot