Saga pattern and event sourcing with kafka

@rafabene e @roanbrasil - https://bit.ly/sagakafka 1
Saga Pattern and Event
Sourcing with Kafka
https://bit.ly/sagakafkaLink
@rafabene
rafael.benevides@oracle.com
@roanbrasil
rmonteiro@astek-canada.ca

@rafabene e @roanbrasil - https://bit.ly/sagakafka
rafael.benevides@gmail.com
@rafabene
apiVersion: oracle/v1
kind: PrincipalSolutionArchitect
metadata:
name: Rafael Benevides
namespace: Oracle Linux team
annotations:
apache/contributor: Apache DeltaSpike
PMC
labels:
developer: Java, NodeJS
hobby: 4x4, drones
spec:
replicas: 1
containers:
image: benevides/rafael:latest
Rafael Benevides

roanbrasil@gmail.com
@roanbrasil
apiVersion: astek-canada/v1
kind: JavaDeveloper
metadata:
name: Roan Brasil Monteiro
namespace: Java Developer team
labels:
developer: Java, NodeJS
hobby: guitar, video game, snorkeling,
soccer
spec:
replicas: 1
containers:
image: brasilmonteiro/roan:latest
Roan Brasil Monteiro

Before we get started…
● I don't use microservices and love my monolith.
● I've started using microservices.
● I've heard about Event Sourcing and Sagas before.
● I do microservices using Event Sourcing at work.
● I've implemented Saga Patterns.
Who are you?

Monolithic architecture
Order
Module
Booking
Module
Payment
Module
Web
UI Database
Browser /
Client
BEWARE: This is
NOT an anti-pattern!

Moving to Microservices
Database
Browser /
Client
Order
Module
Booking
Module
Payment
Module
Web
UI

Moving to Microservices
Database
Browser /
Client
Order
Service
Booking
Service
Payment
Service
Web
UI Database
Database

Microservices Architecture (synchronous calls)
Database
Browser /
Client
Order
Service
Booking
Service
Payment
Service
Web
UI
Database
Database
1) New
Order
2) Room
Reservation
3) ProcessPayment
REST Calls

No ACID transaction that span services
Database
Browser /
Client
Order
Service
Booking
Service
Payment
Service
Web
UI
Database
Database
1) New
Order
2) Room
Reservation
3) ProcessPayment
T1
T2
T3
REST Calls

Exceptions: No room available (synchronous calls)
Database
Browser /
Client
Order
Service
Booking
Service
Web
UI
Database1) New
Order
2) Room
Reservation
3) Exception
happened4) Cancel
order
REST Calls
1:1 services involved
That was easy!

Exceptions: Payment Denied (synchronous calls)
Database
Browser /
Client
Order
Service
Booking
Service
Payment
Service
Web
UI
Database
Database
1) New
Order
2) Room
Reservation
3) ProcessPayment
6) Cancel Booking???6) Cancel
Booking???
REST Calls
4) Exception
happened
5) Cancel
order
● What if I have 10 microservices
to rollback?
● Can you spot the complexity?

Death Star Architectures

Can I use 2PC?
Prepare phase
Database
Browser /
Client
Order
Service
Booking
Service
Payment
Service
Web
UI
Database
Database
1) New
Order
2) Room
Reservation
3) ProcessPayment
Prepare
Prepare
Prepare

Can I use 2PC?
Prepare phase
Database
Browser /
Client
Order
Service
Booking
Service
Payment
Service
Web
UI
Database
Database
1) New
Order
2) Room
Reservation
3) ProcessPayment
Prepared
Prepared
Prepared

Can I use 2PC?
Commit phase
Database
Browser /
Client
Order
Service
Booking
Service
Payment
Service
Web
UI
Database
Database
1) New
Order
2) Room
Reservation
3) ProcessPayment
Commit/
Abort
Commit/
Abort
Commit/
Abort

Can I use 2PC?
Commit phase
Database
Browser /
Client
Order
Service
Booking
Service
Payment
Service
Web
UI
Database
Database
1) New
Order
2) Room
Reservation
3) ProcessPayment
Done
Done
Done
Problems of 2 Phase commit
● Transaction coordinator is the SPF.
● Others have to wait until the slowest resource finish its confirmation.
○ Ex. Credit card payment can take up to 30 seconds to reply.
○ Databases are locked during that time.
○ Can cause deadlocks.
● Reduced throughput due to locks.
● Not supported by many NoSQL databases.

Saga: A solution from 1987
https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf

How does the saga work?
Order
Service
Order
createOrder()
Order{
state = PENDING
}
Order
Service
Order
Order{
state = APPROVED
}
Payment
Service
Account
PaymentInfo{
state = APPROVED
}

How does the rollback work?
Use compensating transactions to "undo" the business logic.
Order
createOrder()
Booking
reserveRoom()
Payment
withdraw()
Order
cancelOrder()
No rooms
available
Booking
cancel
Reservation()
T1 T2 T3
C1 C2
T1 T2 C1
T3 and C2 were not called

Rollback: Payment Denied
Order
Service
Order
createOrder()
Order{
state = PENDING
}
Order
Service
Order
Order{
state = CANCELED
}
Payment
Service
Account
PaymentInfo{
state = DENIED
}

Rollback exception: Payment service unavailable
Order
Service
Order
createOrder()
Order{
state = PENDING
}
Order
Service
Order
Order{
state = PENDING
}
Payment
Service
Account
REST Call
Client and Server must be
always available
Order will be PENDING
forever
We need to go asynchronous

Event Sourcing
(An express explanation)

What is event sourcing?
Architectural style where you:
● Don't save the current state of objects
● Save the events that lead to the current state (ORDER IS IMPORTANT)
● Events happened in the past
● Example:
AccountDeposit
accountNumber: 123456-700
value: 100.50

Creating events to change to state
BankAccountCreated
Id: 123
Owner: Rafael
BankAccount
Events State
BankAccount
Id: 123
Owner: Rafael
Balance: 0
MoneyDeposited
account: 123
value: 100
BankAccount
Id: 123
Owner: Rafael
Balance: 100

Creating events to change to state
MoneyWithdraw
account: 123
value: 60
Events State
BankAccount
Id: 123
Owner: Rafael
Balance: 100
BankAccount
Id: 123
Owner: Rafael
Balance: 40
MoneyWithdraw
account: 123
value: 60
BankAccount
Id: 123
Owner: Rafael
Balance: 40
Event not possible to
be handled

Command Sourcing
(An express explanation)

What is command sourcing?
Architectural style where you:
● Persist commands
● Says what command/intention should be performed
● Wait for replies
● Example:
ExecuteAccountWithdraw
accountNumber:
123456-700
value: 100.50
InsuficientBalanceReply
accountNumber:
123456-700
requestedValue: 100.50

Saga Patterns
● Choreography-based sagas - services collaborate by exchanging domain events
● Orchestration-based sagas - a centralized coordinator sending command
messages to participants, which respond with reply messages
https://eventuate.io/post/eventuate/2020/02/24/why-eventuate.html

Saga Pattern:
Choreography-based

Order
Service
Event Store
createOrder()
Publish
OrderCreatedEvent
Booking
Service
Consume
OrderCreatedEvent
Publish
BookCreatedEvent
Payment
Service
Consume
BookCreatedEvent
Publish
PaymentSuccess
Consume
BookConfirmedEvent
Choreography
Consume
PaymentSuccess
Publish
BookConfirmedEvent

Order
Service
Event Store
createOrder()
Publish
OrderCreatedEvent
Booking
Service
Consume
OrderCreatedEvent
Publish
NoRoomEvent
Payment
Service
Consume
NoRoomEvent
Choreography: No rooms available
Compensating
transaction

Order
Service
Event Store
createOrder()
Publish
OrderCreatedEvent
Booking
Service
Consume
OrderCreatedEvent
Publish
BookCreatedEvent
Payment
Service
Consume
BookCreatedEvent
Publish
PaymentDenied
Consume
BookCanceledEvent
Choreography: Payment denied
Compensating
transactions
Consume
PaymentDenied
Publish
BookCanceledEvent

Advantages and disadvantages of Choreography
● Simple to implement
● Tight coupling - highly dependent upon each other
● The decision logic is distributed - difficulty to maintain and manage the overall
process
● Hard to debug

Event Command Transformation Pattern
Order
Service
NoRoomEvent
BookConfirmedEvent
...
Booking has to know all
events that triggers a
Order action
BookCanceledEvent

Order
Service
Booking
ServiceOrderCreatedEvent BookCommand
Event
Something has
happened in the past
Command
Something has to
happen in the future
Event
Command
Transformation
Where we do
the coupling

Order
Service
Broker
createOrder()
Publish
BookingCommand
Booking
Service
Consume
BookingCommand
Publish
PaymentCommand
Payment
Service
Consume
PaymentCommand
Publish
BookCommand
Consume
OrderCommand

Saga Pattern:
Orchestration-based

Order
Service
Broker
createOrder()
Publish
OrderCreatedEvent
Consume
OrderConfirmation
Command
Orchestration
Orchestrator
Consume
OrderCreatedEvent
Produce
ReservationCommand
Booking
Service
Consume
Reservation
Command
Payment
Service
Publish
ReservationResult
Consume
Payment
Command
Publish
PaymentResult
Consume
ReservationResult
Produce
PaymentCommand
Consume
PaymentResult
Produce
BookingConfirmation
Command
OrderConfirmation
Command
Consume Booking
Confirmation
Command

Order
Service
Broker
createOrder()
Publish
OrderCreatedEvent
Consume
OrderCancelation
Command
Orchestration: Payment Denied
Orchestrator
Consume
OrderCreatedEvent
Produce
ReservationCommand
Booking
Service
Consume
Reservation
Command
Payment
Service
Publish
ReservationResult
Consume
Payment
Command
Publish
PaymentDenied
Consume
ReservationResult
Produce
PaymentCommand
Consume
PaymentDennied
Produce
BookingCancelationC
omand and
OrderCancelation
Command
Consume Booking
Cancelation
Command

Advantages and disadvantages of Orchestration
● Centralized business logic - Easy to understand and maintain.
● Loose coupling as each service consumes only their commands
● Orchestrator is the SPF
● Needs Event -> Command transformation

A note about
Aggregator vs Chain
Patterns

Choreographed
Aggregator
(Parallel)
Chain
(Serial) Order
Service
Booking
Service
Payment
Service
Order
Service
Booking
Service
Payment
Service
Order triggers
compensation
Each service triggers
compensation from
previous transaction
O
rder holds
the
logic
for the
use
case

Orchestrated
Aggregator (Parallel)
Chain (Serial)
Order
Service
Booking
Service Payment
Service
Order
Service
Booking
Service Payment
Service
Orchestrator
Orchestrator
Orchestrator needs to
coordinate states before
calling proper compensation
transactions
Orchestrator flow is
easier to handle
and implement

Aggregator vs Chain Patterns
● Which pattern do you think it is easier
to implement and handle States
● Tip:

Event sourcing with Kafka

Creating an Orchestrator
using Kafka Streams

API to Consume, process and publish to Kafka Topics

Asynchronous chained orchestrated saga

Demo use cases
● Book a room
● Payment offline
● Payment denied
● No rooms available
● Orchestrator offline?

Demo
rafabene/saga-demo

Run them on Linux
VirtualPhysical Private Public
Oracle Linux
Node NodeNode
Master
API ServerDev
Ops
SCM
(Git/Svn)
CI/CD
Automation
Controllers
- Scheduler
- Replication
- Services
- Builds
- Routes
- Deployment
Kubernetes
Ingress Gateway

@rafabene e @roanbrasil - https://bit.ly/sagakafka@rafabene e @roanbrasil
@RAFABENE
@ROANBRASIL

Saga pattern and event sourcing with kafka

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Saga pattern and event sourcing with kafka

Similar to Saga pattern and event sourcing with kafka (20)

More from Roan Brasil Monteiro

More from Roan Brasil Monteiro (7)

Recently uploaded

Recently uploaded (20)

Saga pattern and event sourcing with kafka