Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
@berndruecker
Lost in transaction?
Strategies to manage consistency
in distributed systems
@berndruecker
Order
Order
Line Item
Customer
Customer
Address
orderId
customerId
chargeId
…
customerId
…
productId
price
…
Charge
Charge...
Order
Order
Line Item
orderId
customerId
chargeId
…
productId
price
… Order
Charge
Charge
chargeId
amount
paymentProvider
...
A
C
I
D
Atomicity
Consistency
Isolation
Durability
@berndruecker
Aggregates = Consistency Boundaries
Order
Order
Line Item
orderId
customerId
chargeId
…
productId
price
… Order
Charge
Cha...
Aggregates = Consistency Boundaries
Order
Order
Line Item
orderId
customerId
chargeId
…
productId
price
… Order
Charge
Cha...
Aggregates = Consistency Boundaries
Order
Order
Line Item
orderId
customerId
chargeId
…
productId
price
… Order
Charge
Cha...
Boundaries need to be designed carefully
Charge
chargeId
amount
paymentProvider
transactionId
…
Order
Order
Line Item
orde...
But no implicit constraints!
Order
Order
Line Item
orderId
customerId
chargeId
…
productId
price
… Order
Charge
Charge
cha...
There is two-phase commit (XA)!!
TX
Coordinator
Resource
Managers
Prepare
Phase
Commit
Phase
@berndruecker
Pat Helland
“
Distributed Systems Guru
Worked at Amazon,
Microsoft & Salesforce
@berndruecker
Pat Helland
Grown-Ups Don’t Use
Distributed Transactions
“
Distributed Systems Guru
Worked at Amazon,
Microsoft & Salesfor...
Starbucks does not use two phase commit
https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html
Photo by...
Eric Brewer
Atomicity
Consistency
Isolation
Durability
http://pld.cs.luc.edu/courses/353/spr11/notes/brewer_keynote.pdf @b...
That means
Do A
Do B
Temporarily
inconsistent
Eventually
consistent
again
t
Consistent
Local
ACID
Local
ACID
1 aggregate
1...
Youmightknowthisfrom:
Do A
Do B
Temporarily
inconsistent
Eventually
consistent
again
t
Consistent
Photo by Gerhard51, avai...
„Building on Quicksand“ Paper
A
C
I
D
2.0Pat Helland
@berndruecker
Associative
Commutative
Idempotent
Distributed
2.0
(a + b) + c = a + (b + c)
a + b = b + a
f(x) = f( f(x) )
„Building on Q...
Distributed
@berndruecker
Distributed systems
@berndruecker
Network problems
Credit
Card
Payment
charge
@berndruecker
Strategy: retry
Credit
Card
Payment
Charge Credit Card
cardNumber
amount
Charge Credit Card
cardNumber
amount
transactionI...
Photo by pixabay, available under Creative Commons CC0 1.0 license.
@berndruecker
Requirement: Idempotency of services!
Photo by pixabay, available under Creative Commons CC0 1.0 license.
@berndruecker
Requirement: Idempotency of services!
Photo by Chr.Späth, available under Public Domain.
@berndruecker
Distributed systems
@berndruecker
It is impossible to
differentiate certain
failure scenarios:
Independant of
communication style!
Service
Provider
Client
@...
Strategy: Cleanup
Credit
Card
Payment
charge
Make sure it is
not charged!
Cancel charge
cardNumber
amount
transactionId
Ra...
Distributed systems
@berndruecker
Some communication challenges
require state.
@berndruecker
Strategy: Stateful retry
Credit
Card
Payment
charge
@berndruecker
Strategy: Stateful retry
Credit
Card
Payment
charge
Make sure it is
not charged!
@berndruecker
Warning:
Contains Opinion
Berlin, Germany
bernd.ruecker@camunda.com
@berndruecker
Bernd Ruecker
Co-founder and
Technologist of
Camunda
Payment
Stateful retry
Credit
Card
REST
@berndruecker
Stateful retry & cleanup
Credit
Card
Payment
REST
Cancel
charge
@berndruecker
Architecture
Infrastructure
Application
Domain
Workflow
Engine
@berndruecker
Architecture
Infrastructure
Application
Domain
Workflow
Engine
@berndruecker
Lightweight
OSS workflow
engine
Live hacking
https://github.com/flowing/flowing-retail/tree/master/rest
@berndruecker
Embedded Engine Example (Java)
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
@...
A relatively common pattern
Service
(e.g. Go)
Rabbit
RDMS
1. Receive
4. Send additional events
2. Business Logic
3. Send
r...
„Can this handle 15k
requests per second?“
@berndruecker
„Yes.“
@berndruecker
@berndruecker
@berndruecker
Compensation – the classical example
Saga
book
hotel
book
car
book
flight
cancel
hotel
cancel
car
1. 2. 3.
5.6.
In case of...
2 alterntive approaches: choreography & orchestration
@berndruecker
Event-driven choreography
Hotel
Flight
Car
Trip
Trip
booked
Flight
booked
Trip
requested
Hotel
booked
Car
booked
Request
t...
Event-driven choreography
Hotel
Flight
Car
Trip
Trip
failed
Trip
requested
Hotel
booked
Car
booked
Request
trip
Flight
fai...
The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
...
The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
...
The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
...
If your transaction involves 2 to 4 steps,
choreography might be a very good fit.
However, this approach can rapidly becom...
Implementing changes in the process
Hotel
Flight
Car
Trip
Trip
failed
Trip
requested
Hotel
booked
Car
booked
Request
trip
...
Implementing changes in the process
Hotel
Flight
Car
Trip
Trip
failed
Trip
requested
Hotel
booked
Car
booked
Request
trip
...
What we wanted
Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0
@bernd...
Orchestration
Hotel
Flight
Car
Trip
Trip
booked
Request
trip
Book
hotel
Hotel
booked
Car
booked
Flight
booked
Book
car
Boo...
Orchestration
Hotel
Flight
Car
Trip
Trip
booked
Request
trip
Book
hotel
Hotel
booked
Car
booked
Flight
booked
Book
car
Boo...
Describe orchestration with BPMN
Trip
Trip
booked
Request
trip
@berndruecker
The workflow is domain logic as part of the service
Trip
@berndruecker
The workflow is domain logic as part of the service
Trip
Payment
Payment could be one
step in the Trip Saga
@berndruecker
Caitie McCaffrey | @caitie
@berndruecker
Graphical models?
@berndruecker
Clemens Vasters
Architect at Microsoft
http://vasters.com/archive/Sagas.html
@berndruecker
Clemens Vasters
Architect at Microsoft
http://vasters.com/archive/Sagas.html
@berndruecker
Clemens Vasters
Architect at Microsoft
http://vasters.com/archive/Sagas.html
@berndruecker
BPMN
Business Process
Model and Notation
ISO Standard
@berndruecker
Living documentation for long-running behaviour
@berndruecker
Visual HTML reports for test cases
@berndruecker
BizDevOps
@berndruecker
Fancy a DSL? Just do it!
https://github.com/berndruecker/flowing-trip-booking-saga
The visual get
auto-generated…
@berndru...
Thoughts on the state machine | workflow engine market
@berndruecker
Thoughts on the state machine | workflow engine market
OSS Workflow or
Orchestration Engines
Stack Vendors,
Pure Play BPMS...
Does it support stateful operations?
Does it support the necessary flow logic?
Does it support BizDevOps?
Does it scale?
@...
My personal pro-tip for a shortlist ;-)
OSS Workflow or
Orchestration Engines
Stack Vendors,
Pure Play BPMS
Low Code Platf...
Recap
• Aggregates = Consistency boundaries
• Grown ups don‘t use distributed transactions
but eventual consistency
• Idem...
Thank you!
@berndruecker
mail@berndruecker.io
@berndruecker
https://berndruecker.io
https://medium.com/berndruecker
https://github.com/berndruecker...
Upcoming SlideShare
Loading in …5
×

2019 - Lost in transaction

Slides from my talk held in 2019, e.g. at QCon, MicroXchg, JPoint Moscow or JavaLand...

See https://berndruecker.io/lost-in-transaction/ for details

  • Be the first to comment

2019 - Lost in transaction

  1. 1. @berndruecker Lost in transaction? Strategies to manage consistency in distributed systems
  2. 2. @berndruecker
  3. 3. Order Order Line Item Customer Customer Address orderId customerId chargeId … customerId … productId price … Charge Charge Order chargeId amount paymentProvider transactionId … Aggregate Entity Value Object Reference by identity @berndruecker
  4. 4. Order Order Line Item orderId customerId chargeId … productId price … Order Charge Charge chargeId amount paymentProvider transactionId … Create Charge Mark order as paid All or nothing + @berndruecker
  5. 5. A C I D Atomicity Consistency Isolation Durability @berndruecker
  6. 6. Aggregates = Consistency Boundaries Order Order Line Item orderId customerId chargeId … productId price … Order Charge Charge chargeId amount paymentProvider transactionId … You can do ACID within here Or here But not a joined ACID transaction! @berndruecker
  7. 7. Aggregates = Consistency Boundaries Order Order Line Item orderId customerId chargeId … productId price … Order Charge Charge chargeId amount paymentProvider transactionId … Own DB / autonomous technology decisions API-driven Scalability … @berndruecker
  8. 8. Aggregates = Consistency Boundaries Order Order Line Item orderId customerId chargeId … productId price … Order Charge Charge chargeId amount paymentProvider transactionId … Own DB / autonomous technology decisions API-driven Scalability … @berndruecker
  9. 9. Boundaries need to be designed carefully Charge chargeId amount paymentProvider transactionId … Order Order Line Item orderId customerId chargeId … productId price …Order @berndruecker
  10. 10. But no implicit constraints! Order Order Line Item orderId customerId chargeId … productId price … Order Charge Charge chargeId amount paymentProvider transactionId … Joined DB transaciton @berndruecker
  11. 11. There is two-phase commit (XA)!! TX Coordinator Resource Managers Prepare Phase Commit Phase @berndruecker
  12. 12. Pat Helland “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce @berndruecker
  13. 13. Pat Helland Grown-Ups Don’t Use Distributed Transactions “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce @berndruecker
  14. 14. Starbucks does not use two phase commit https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html Photo by John Ingle @berndruecker
  15. 15. Eric Brewer Atomicity Consistency Isolation Durability http://pld.cs.luc.edu/courses/353/spr11/notes/brewer_keynote.pdf @berndruecker
  16. 16. That means Do A Do B Temporarily inconsistent Eventually consistent again t Consistent Local ACID Local ACID 1 aggregate 1 (micro-)service 1 program 1 resource Violates „I“ of ACID @berndruecker
  17. 17. Youmightknowthisfrom: Do A Do B Temporarily inconsistent Eventually consistent again t Consistent Photo by Gerhard51, available under Creative Commons CC0 1.0 license. @berndruecker
  18. 18. „Building on Quicksand“ Paper A C I D 2.0Pat Helland @berndruecker
  19. 19. Associative Commutative Idempotent Distributed 2.0 (a + b) + c = a + (b + c) a + b = b + a f(x) = f( f(x) ) „Building on Quicksand“ Paper Pat Helland @berndruecker
  20. 20. Distributed @berndruecker
  21. 21. Distributed systems @berndruecker
  22. 22. Network problems Credit Card Payment charge @berndruecker
  23. 23. Strategy: retry Credit Card Payment Charge Credit Card cardNumber amount Charge Credit Card cardNumber amount transactionId Not idempotent Idempotent has to be idempotent charge @berndruecker
  24. 24. Photo by pixabay, available under Creative Commons CC0 1.0 license. @berndruecker
  25. 25. Requirement: Idempotency of services! Photo by pixabay, available under Creative Commons CC0 1.0 license. @berndruecker
  26. 26. Requirement: Idempotency of services! Photo by Chr.Späth, available under Public Domain. @berndruecker
  27. 27. Distributed systems @berndruecker
  28. 28. It is impossible to differentiate certain failure scenarios: Independant of communication style! Service Provider Client @berndruecker
  29. 29. Strategy: Cleanup Credit Card Payment charge Make sure it is not charged! Cancel charge cardNumber amount transactionId Raise payment failed @berndruecker
  30. 30. Distributed systems @berndruecker
  31. 31. Some communication challenges require state. @berndruecker
  32. 32. Strategy: Stateful retry Credit Card Payment charge @berndruecker
  33. 33. Strategy: Stateful retry Credit Card Payment charge Make sure it is not charged! @berndruecker
  34. 34. Warning: Contains Opinion
  35. 35. Berlin, Germany bernd.ruecker@camunda.com @berndruecker Bernd Ruecker Co-founder and Technologist of Camunda
  36. 36. Payment Stateful retry Credit Card REST @berndruecker
  37. 37. Stateful retry & cleanup Credit Card Payment REST Cancel charge @berndruecker
  38. 38. Architecture Infrastructure Application Domain Workflow Engine @berndruecker
  39. 39. Architecture Infrastructure Application Domain Workflow Engine @berndruecker Lightweight OSS workflow engine
  40. 40. Live hacking https://github.com/flowing/flowing-retail/tree/master/rest @berndruecker
  41. 41. Embedded Engine Example (Java) https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91 @berndruecker
  42. 42. A relatively common pattern Service (e.g. Go) Rabbit RDMS 1. Receive 4. Send additional events 2. Business Logic 3. Send response? ACK @berndruecker
  43. 43. „Can this handle 15k requests per second?“ @berndruecker
  44. 44. „Yes.“ @berndruecker
  45. 45. @berndruecker
  46. 46. @berndruecker
  47. 47. Compensation – the classical example Saga book hotel book car book flight cancel hotel cancel car 1. 2. 3. 5.6. In case of failure trigger compensations book trip
  48. 48. 2 alterntive approaches: choreography & orchestration @berndruecker
  49. 49. Event-driven choreography Hotel Flight Car Trip Trip booked Flight booked Trip requested Hotel booked Car booked Request trip @berndruecker
  50. 50. Event-driven choreography Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled Perform undo (cancel car booking) Perform undo (cancel hotel) @berndruecker
  51. 51. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  52. 52. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  53. 53. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  54. 54. If your transaction involves 2 to 4 steps, choreography might be a very good fit. However, this approach can rapidly become confusing if you keep adding extra steps in your transaction as it is difficult to track which services listen to which events. Moreover, it also might add a cyclic dependency between services as they have to subscribe to one another’s events. Denis Rosa Couchbase https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part/ @berndruecker
  55. 55. Implementing changes in the process Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first! @berndruecker
  56. 56. Implementing changes in the process Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled You have to adjust all services and redeploy at the same time! We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first! @berndruecker
  57. 57. What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0 @berndruecker
  58. 58. Orchestration Hotel Flight Car Trip Trip booked Request trip Book hotel Hotel booked Car booked Flight booked Book car Book flight @berndruecker
  59. 59. Orchestration Hotel Flight Car Trip Trip booked Request trip Book hotel Hotel booked Car booked Flight booked Book car Book flight We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first! You have to adjust one service and redeploy only this one! @berndruecker
  60. 60. Describe orchestration with BPMN Trip Trip booked Request trip @berndruecker
  61. 61. The workflow is domain logic as part of the service Trip @berndruecker
  62. 62. The workflow is domain logic as part of the service Trip Payment Payment could be one step in the Trip Saga @berndruecker
  63. 63. Caitie McCaffrey | @caitie @berndruecker
  64. 64. Graphical models? @berndruecker
  65. 65. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html @berndruecker
  66. 66. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html @berndruecker
  67. 67. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html @berndruecker
  68. 68. BPMN Business Process Model and Notation ISO Standard @berndruecker
  69. 69. Living documentation for long-running behaviour @berndruecker
  70. 70. Visual HTML reports for test cases @berndruecker
  71. 71. BizDevOps @berndruecker
  72. 72. Fancy a DSL? Just do it! https://github.com/berndruecker/flowing-trip-booking-saga The visual get auto-generated… @berndruecker
  73. 73. Thoughts on the state machine | workflow engine market @berndruecker
  74. 74. Thoughts on the state machine | workflow engine market OSS Workflow or Orchestration Engines Stack Vendors, Pure Play BPMS Low Code Platforms Homegrown frameworks to scratch an itch Integration Frameworks Cloud Offerings Uber, Netflix, AirBnb, ING, … AWS Step Functions, Azure Durable Functions, … Camunda, Zeebe, jBPM, Activiti, Mistral, … PEGA, IBM, SAG, … Apache Airflow, Spring Data Flow, … Apache Camel, Balerina, … Data Pipelines @berndruecker
  75. 75. Does it support stateful operations? Does it support the necessary flow logic? Does it support BizDevOps? Does it scale? @berndruecker
  76. 76. My personal pro-tip for a shortlist ;-) OSS Workflow or Orchestration Engines Stack Vendors, Pure Play BPMS Low Code Platforms Homegrown frameworks to scratch an itch Integration Frameworks Cloud Offerings Data Pipelines Camunda & Zeebe @berndruecker
  77. 77. Recap • Aggregates = Consistency boundaries • Grown ups don‘t use distributed transactions but eventual consistency • Idempotency is super important • Some consistency challenges require state • Stateful retry & cleanup • Saga / Compensation @berndruecker
  78. 78. Thank you! @berndruecker
  79. 79. mail@berndruecker.io @berndruecker https://berndruecker.io https://medium.com/berndruecker https://github.com/berndruecker https://www.infoq.com/articles/events- workflow-automation Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html https://thenewstack.io/5-workflow-automation- use-cases-you-might-not-have-considered/

×