Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Saga pattern and event sourcing with kafka

Trilha NodeJS com meu amigo Rafael Benevides

  • Be the first to comment

Saga pattern and event sourcing with kafka

  1. 1. @rafabene e @roanbrasil - https://bit.ly/sagakafka 1 Saga Pattern and Event Sourcing with Kafka https://bit.ly/sagakafkaLink @rafabene rafael.benevides@oracle.com @roanbrasil rmonteiro@astek-canada.ca
  2. 2. @rafabene e @roanbrasil - https://bit.ly/sagakafka rafael.benevides@gmail.com @rafabene apiVersion: oracle/v1 kind: PrincipalSolutionArchitect metadata: name: Rafael Benevides namespace: Oracle Linux team annotations: apache/contributor: Apache DeltaSpike PMC labels: developer: Java, NodeJS hobby: 4x4, drones spec: replicas: 1 containers: image: benevides/rafael:latest Rafael Benevides
  3. 3. @rafabene e @roanbrasil - https://bit.ly/sagakafka roanbrasil@gmail.com @roanbrasil apiVersion: astek-canada/v1 kind: JavaDeveloper metadata: name: Roan Brasil Monteiro namespace: Java Developer team labels: developer: Java, NodeJS hobby: guitar, video game, snorkeling, soccer spec: replicas: 1 containers: image: brasilmonteiro/roan:latest Roan Brasil Monteiro
  4. 4. @rafabene e @roanbrasil - https://bit.ly/sagakafka Before we get started… ● I don't use microservices and love my monolith. ● I've started using microservices. ● I've heard about Event Sourcing and Sagas before. ● I do microservices using Event Sourcing at work. ● I've implemented Saga Patterns. Who are you?
  5. 5. @rafabene e @roanbrasil - https://bit.ly/sagakafka
  6. 6. @rafabene e @roanbrasil - https://bit.ly/sagakafka Monolithic architecture Order Module Booking Module Payment Module Web UI Database Browser / Client BEWARE: This is NOT an anti-pattern!
  7. 7. @rafabene e @roanbrasil - https://bit.ly/sagakafka Moving to Microservices Database Browser / Client Order Module Booking Module Payment Module Web UI
  8. 8. @rafabene e @roanbrasil - https://bit.ly/sagakafka Moving to Microservices Database Browser / Client Order Service Booking Service Payment Service Web UI Database Database
  9. 9. @rafabene e @roanbrasil - https://bit.ly/sagakafka Microservices Architecture (synchronous calls) Database Browser / Client Order Service Booking Service Payment Service Web UI Database Database 1) New Order 2) Room Reservation 3) ProcessPayment REST Calls
  10. 10. @rafabene e @roanbrasil - https://bit.ly/sagakafka No ACID transaction that span services Database Browser / Client Order Service Booking Service Payment Service Web UI Database Database 1) New Order 2) Room Reservation 3) ProcessPayment T1 T2 T3 REST Calls
  11. 11. @rafabene e @roanbrasil - https://bit.ly/sagakafka Exceptions: No room available (synchronous calls) Database Browser / Client Order Service Booking Service Web UI Database1) New Order 2) Room Reservation 3) Exception happened4) Cancel order REST Calls 1:1 services involved That was easy!
  12. 12. @rafabene e @roanbrasil - https://bit.ly/sagakafka Exceptions: Payment Denied (synchronous calls) Database Browser / Client Order Service Booking Service Payment Service Web UI Database Database 1) New Order 2) Room Reservation 3) ProcessPayment 6) Cancel Booking???6) Cancel Booking??? REST Calls 4) Exception happened 5) Cancel order ● What if I have 10 microservices to rollback? ● Can you spot the complexity?
  13. 13. @rafabene e @roanbrasil - https://bit.ly/sagakafka Death Star Architectures
  14. 14. @rafabene e @roanbrasil - https://bit.ly/sagakafka Can I use 2PC? Prepare phase Database Browser / Client Order Service Booking Service Payment Service Web UI Database Database 1) New Order 2) Room Reservation 3) ProcessPayment Prepare Prepare Prepare
  15. 15. @rafabene e @roanbrasil - https://bit.ly/sagakafka Can I use 2PC? Prepare phase Database Browser / Client Order Service Booking Service Payment Service Web UI Database Database 1) New Order 2) Room Reservation 3) ProcessPayment Prepared Prepared Prepared
  16. 16. @rafabene e @roanbrasil - https://bit.ly/sagakafka Can I use 2PC? Commit phase Database Browser / Client Order Service Booking Service Payment Service Web UI Database Database 1) New Order 2) Room Reservation 3) ProcessPayment Commit/ Abort Commit/ Abort Commit/ Abort
  17. 17. @rafabene e @roanbrasil - https://bit.ly/sagakafka Can I use 2PC? Commit phase Database Browser / Client Order Service Booking Service Payment Service Web UI Database Database 1) New Order 2) Room Reservation 3) ProcessPayment Done Done Done Problems of 2 Phase commit ● Transaction coordinator is the SPF. ● Others have to wait until the slowest resource finish its confirmation. ○ Ex. Credit card payment can take up to 30 seconds to reply. ○ Databases are locked during that time. ○ Can cause deadlocks. ● Reduced throughput due to locks. ● Not supported by many NoSQL databases.
  18. 18. @rafabene e @roanbrasil - https://bit.ly/sagakafka Saga: A solution from 1987 https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf
  19. 19. @rafabene e @roanbrasil - https://bit.ly/sagakafka How does the saga work? Order Service Order createOrder() Order{ state = PENDING } Order Service Order Order{ state = APPROVED } Payment Service Account PaymentInfo{ state = APPROVED }
  20. 20. @rafabene e @roanbrasil - https://bit.ly/sagakafka How does the rollback work? Use compensating transactions to "undo" the business logic. Order createOrder() Booking reserveRoom() Payment withdraw() Order cancelOrder() No rooms available Booking cancel Reservation() T1 T2 T3 C1 C2 T1 T2 C1 T3 and C2 were not called
  21. 21. @rafabene e @roanbrasil - https://bit.ly/sagakafka Rollback: Payment Denied Order Service Order createOrder() Order{ state = PENDING } Order Service Order Order{ state = CANCELED } Payment Service Account PaymentInfo{ state = DENIED }
  22. 22. @rafabene e @roanbrasil - https://bit.ly/sagakafka Rollback exception: Payment service unavailable Order Service Order createOrder() Order{ state = PENDING } Order Service Order Order{ state = PENDING } Payment Service Account REST Call Client and Server must be always available Order will be PENDING forever We need to go asynchronous
  23. 23. @rafabene e @roanbrasil - https://bit.ly/sagakafka Event Sourcing (An express explanation)
  24. 24. @rafabene e @roanbrasil - https://bit.ly/sagakafka What is event sourcing? Architectural style where you: ● Don't save the current state of objects ● Save the events that lead to the current state (ORDER IS IMPORTANT) ● Events happened in the past ● Example: AccountDeposit accountNumber: 123456-700 value: 100.50
  25. 25. @rafabene e @roanbrasil - https://bit.ly/sagakafka Creating events to change to state BankAccountCreated Id: 123 Owner: Rafael BankAccount Events State BankAccount Id: 123 Owner: Rafael Balance: 0 MoneyDeposited account: 123 value: 100 BankAccount Id: 123 Owner: Rafael Balance: 100
  26. 26. @rafabene e @roanbrasil - https://bit.ly/sagakafka Creating events to change to state MoneyWithdraw account: 123 value: 60 Events State BankAccount Id: 123 Owner: Rafael Balance: 100 BankAccount Id: 123 Owner: Rafael Balance: 40 MoneyWithdraw account: 123 value: 60 BankAccount Id: 123 Owner: Rafael Balance: 40 Event not possible to be handled
  27. 27. @rafabene e @roanbrasil - https://bit.ly/sagakafka Command Sourcing (An express explanation)
  28. 28. @rafabene e @roanbrasil - https://bit.ly/sagakafka What is command sourcing? Architectural style where you: ● Persist commands ● Says what command/intention should be performed ● Wait for replies ● Example: ExecuteAccountWithdraw accountNumber: 123456-700 value: 100.50 InsuficientBalanceReply accountNumber: 123456-700 requestedValue: 100.50
  29. 29. @rafabene e @roanbrasil - https://bit.ly/sagakafka Saga Patterns ● Choreography-based sagas - services collaborate by exchanging domain events ● Orchestration-based sagas - a centralized coordinator sending command messages to participants, which respond with reply messages https://eventuate.io/post/eventuate/2020/02/24/why-eventuate.html
  30. 30. @rafabene e @roanbrasil - https://bit.ly/sagakafka Saga Pattern: Choreography-based
  31. 31. @rafabene e @roanbrasil - https://bit.ly/sagakafka Order Service Event Store createOrder() Publish OrderCreatedEvent Booking Service Consume OrderCreatedEvent Publish BookCreatedEvent Payment Service Consume BookCreatedEvent Publish PaymentSuccess Consume BookConfirmedEvent Choreography Consume PaymentSuccess Publish BookConfirmedEvent
  32. 32. @rafabene e @roanbrasil - https://bit.ly/sagakafka Order Service Event Store createOrder() Publish OrderCreatedEvent Booking Service Consume OrderCreatedEvent Publish NoRoomEvent Payment Service Consume NoRoomEvent Choreography: No rooms available Compensating transaction
  33. 33. @rafabene e @roanbrasil - https://bit.ly/sagakafka Order Service Event Store createOrder() Publish OrderCreatedEvent Booking Service Consume OrderCreatedEvent Publish BookCreatedEvent Payment Service Consume BookCreatedEvent Publish PaymentDenied Consume BookCanceledEvent Choreography: Payment denied Compensating transactions Consume PaymentDenied Publish BookCanceledEvent
  34. 34. @rafabene e @roanbrasil - https://bit.ly/sagakafka Advantages and disadvantages of Choreography ● Simple to implement ● Tight coupling - highly dependent upon each other ● The decision logic is distributed - difficulty to maintain and manage the overall process ● Hard to debug
  35. 35. @rafabene e @roanbrasil - https://bit.ly/sagakafka Event Command Transformation Pattern Order Service NoRoomEvent BookConfirmedEvent ... Booking has to know all events that triggers a Order action BookCanceledEvent
  36. 36. @rafabene e @roanbrasil - https://bit.ly/sagakafka Event Command Transformation Pattern Order Service Booking ServiceOrderCreatedEvent BookCommand Event Something has happened in the past Command Something has to happen in the future Event Command Transformation Where we do the coupling
  37. 37. @rafabene e @roanbrasil - https://bit.ly/sagakafka Order Service Broker createOrder() Publish BookingCommand Booking Service Consume BookingCommand Publish PaymentCommand Payment Service Consume PaymentCommand Publish BookCommand Consume OrderCommand Event Command Transformation Pattern
  38. 38. @rafabene e @roanbrasil - https://bit.ly/sagakafka Saga Pattern: Orchestration-based
  39. 39. @rafabene e @roanbrasil - https://bit.ly/sagakafka Order Service Broker createOrder() Publish OrderCreatedEvent Consume OrderConfirmation Command Orchestration Orchestrator Consume OrderCreatedEvent Produce ReservationCommand Booking Service Consume Reservation Command Payment Service Publish ReservationResult Consume Payment Command Publish PaymentResult Consume ReservationResult Produce PaymentCommand Consume PaymentResult Produce BookingConfirmation Command OrderConfirmation Command Consume Booking Confirmation Command
  40. 40. @rafabene e @roanbrasil - https://bit.ly/sagakafka Order Service Broker createOrder() Publish OrderCreatedEvent Consume OrderCancelation Command Orchestration: Payment Denied Orchestrator Consume OrderCreatedEvent Produce ReservationCommand Booking Service Consume Reservation Command Payment Service Publish ReservationResult Consume Payment Command Publish PaymentDenied Consume ReservationResult Produce PaymentCommand Consume PaymentDennied Produce BookingCancelationC omand and OrderCancelation Command Consume Booking Cancelation Command
  41. 41. @rafabene e @roanbrasil - https://bit.ly/sagakafka Advantages and disadvantages of Orchestration ● Centralized business logic - Easy to understand and maintain. ● Loose coupling as each service consumes only their commands ● Orchestrator is the SPF ● Needs Event -> Command transformation
  42. 42. @rafabene e @roanbrasil - https://bit.ly/sagakafka A note about Aggregator vs Chain Patterns
  43. 43. @rafabene e @roanbrasil - https://bit.ly/sagakafka Choreographed Aggregator (Parallel) Chain (Serial) Order Service Booking Service Payment Service Order Service Booking Service Payment Service Order triggers compensation Each service triggers compensation from previous transaction O rder holds the logic for the use case
  44. 44. @rafabene e @roanbrasil - https://bit.ly/sagakafka Orchestrated Aggregator (Parallel) Chain (Serial) Order Service Booking Service Payment Service Order Service Booking Service Payment Service Orchestrator Orchestrator Orchestrator needs to coordinate states before calling proper compensation transactions Orchestrator flow is easier to handle and implement
  45. 45. @rafabene e @roanbrasil - https://bit.ly/sagakafka Aggregator vs Chain Patterns ● Which pattern do you think it is easier to implement and handle States ● Tip:
  46. 46. @rafabene e @roanbrasil - https://bit.ly/sagakafka Event sourcing with Kafka
  47. 47. @rafabene e @roanbrasil - https://bit.ly/sagakafka Creating an Orchestrator using Kafka Streams
  48. 48. @rafabene e @roanbrasil - https://bit.ly/sagakafka API to Consume, process and publish to Kafka Topics
  49. 49. @rafabene e @roanbrasil - https://bit.ly/sagakafka Asynchronous chained orchestrated saga
  50. 50. @rafabene e @roanbrasil - https://bit.ly/sagakafka Demo use cases ● Book a room ● Payment offline ● Payment denied ● No rooms available ● Orchestrator offline?
  51. 51. @rafabene e @roanbrasil - https://bit.ly/sagakafka Demo rafabene/saga-demo
  52. 52. @rafabene e @roanbrasil - https://bit.ly/sagakafka Run them on Linux VirtualPhysical Private Public Oracle Linux Node NodeNode Master API ServerDev Ops SCM (Git/Svn) CI/CD Automation Controllers - Scheduler - Replication - Services - Builds - Routes - Deployment Kubernetes Ingress Gateway
  53. 53. @rafabene e @roanbrasil - https://bit.ly/sagakafka
  54. 54. @rafabene e @roanbrasil - https://bit.ly/sagakafka@rafabene e @roanbrasil @RAFABENE @ROANBRASIL

×