Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Managing Data in
Microservices
Randy Shoup
VP Engineering, Stitch Fix
@randyshoup
linkedin.com/in/randyshoup
First Law of Distributed Object
Design:
Don’t distribute your objects!
-- Martin Fowler
Prerequisites for
Microservices
• Domain Maturity: Know enough about your domain to
decompose it cleanly
• Organizational ...
Microservices
• Single-purpose
• Simple, well-defined interface
• Modular and independent
A
C D E
B
Microservices are nothing
more than SOA done properly.
-- me
Microservices
• Single-purpose
• Simple, well-defined interface
• Modular and independent
• Isolated persistence (!)
A
C D...
Microservice
Persistence
• Approach 1: Operate your own data store
o Store to your own instance(s) of {Postgres, MySQL, et...
Extracting
Microservices
• Problem: Monolithic shared DB
• Clients
• Shipments
• Items
• Styles, SKUs
• Warehouses
• etc.
...
Extracting
Microservices
• Decouple applications / services from shared DB
• Clients
• Shipments
• Items
• Styles, SKUs
• ...
Extracting
Microservices
• Decouple applications / services from shared DB
Styling app Warehouse app
core_item
core_sku
co...
Extracting
Microservices
• Step 1: Create a service
Styling app Warehouse app
core_item
core_sku
core_client
client-service
Extracting
Microservices
• Step 2: Applications use the service
Styling app Warehouse app
core_item
core_sku
core_client
c...
Extracting
Microservices
• Step 3: Move data to private database
Styling app Warehouse app
core_item
core_sku
client-servi...
Extracting
Microservices
• Step 4: Rinse and Repeat
Styling app Warehouse app
core_sku
client-service
core_client
item-ser...
Extracting
Microservices
• Step 4: Rinse and Repeat
Styling app Warehouse app
client-service
core_client
item-service
core...
Extracting
Microservices
• Step 4: Rinse and Repeat
Styling app Warehouse app
client-service
core_client
item-service
core...
Microservice Techniques:
Shared Data
• Problem
o Monolithic database makes it easy to leverage shared data
o Where does sh...
Microservice Techniques:
Shared Data
• Approach 1: Synchronous Lookup
o Customer service owns customer data
o Order servic...
Microservice Techniques:
Shared Data
• Approach 2: Async event + local cache
o Customer service owns customer data
o Custo...
Microservice Techniques:
Shared Data
• Approach 3: Shared metadata library
o Read-only metadata, basically immutable
o E.g...
Events as
First-Class Construct
• “A significant change in state”
o Statement that some interesting thing occurred
o 0 | 1...
Microservices
and Events
• Events are a first-class part of the interface
• A service interface includes
o Synchronous req...
Microservice Techniques:
Joins
• Problem
o Monolithic database makes joins very easy
o Splitting the data into separate se...
Microservice Techniques:
Joins
• Approach 1: Service that “Materializes the View”
o Listen to events from item-service, ev...
Microservice Techniques:
Joins
• Many common systems do this
o Most NoSQL approaches
o “Materialized view” in database sys...
Microservice Techniques:
Joins
• Approach 2: Join in Application / Client
o Get a single customer from customer-service
o ...
Microservice Techniques:
Joins
• Many common systems do this
o Web application “mashup”
Microservice Techniques:
Workflows and Sagas
• Problem
o Monolithic database makes transactions across multiple entities e...
Microservice Techniques:
Workflows and Sagas
• Transaction  Saga
o Model the transaction as a state machine of atomic eve...
Microservice Techniques:
Workflows and Sagas
• Many common systems do this
o Payment processing flow
o Most approval workf...
Microservice Techniques:
Workflows and Sagas
• Ideal use for Functions as a Service (“Serverless”)
o Very lightweight logi...
Thanks!
• @randyshoup
• linkedin.com/in/randyshoup
Upcoming SlideShare
Loading in …5
×

Managing Data in Microservices

From a talk at the SF CTO Summit 2017 (https://www.ctoconnection.com/summits/sf2017), these slides cover the speaker's experience at Stitch Fix with managing data in a microservices environment. Areas include:
* Breaking up a monolithic database into services
* Using events as a first-class part of your architecture
* Sharing data among microservices
* Handling "joins" among microservices
* Simulating "transactions" among microservices using the Saga pattern

Managing Data in Microservices

  1. 1. Managing Data in Microservices Randy Shoup VP Engineering, Stitch Fix @randyshoup linkedin.com/in/randyshoup
  2. 2. First Law of Distributed Object Design: Don’t distribute your objects! -- Martin Fowler
  3. 3. Prerequisites for Microservices • Domain Maturity: Know enough about your domain to decompose it cleanly • Organizational Maturity: Small teams with well-defined areas of responsibility • Process Maturity: Automated Testing and Continuous Delivery • Operational Maturity: Monitoring, alerting, oncall
  4. 4. Microservices • Single-purpose • Simple, well-defined interface • Modular and independent A C D E B
  5. 5. Microservices are nothing more than SOA done properly. -- me
  6. 6. Microservices • Single-purpose • Simple, well-defined interface • Modular and independent • Isolated persistence (!) A C D E B
  7. 7. Microservice Persistence • Approach 1: Operate your own data store o Store to your own instance(s) of {Postgres, MySQL, etc.}, owned and operated by the service team • Approach 2: Use a persistence service o Store to your own table(s) in {Dynamo, RDS, Spanner, etc.}, operated as a service by another team or by a third-party provider o Isolated from all other users of the service •  Only external access to data store is through published service interface
  8. 8. Extracting Microservices • Problem: Monolithic shared DB • Clients • Shipments • Items • Styles, SKUs • Warehouses • etc. stitchfix.com Styling app Warehouse app Merch app CS app Logistics app Payments service Profile service
  9. 9. Extracting Microservices • Decouple applications / services from shared DB • Clients • Shipments • Items • Styles, SKUs • Warehouses • etc. stitchfix.com Styling app Warehouse app Merch app CS app Logistics app Payments service Profile service
  10. 10. Extracting Microservices • Decouple applications / services from shared DB Styling app Warehouse app core_item core_sku core_client
  11. 11. Extracting Microservices • Step 1: Create a service Styling app Warehouse app core_item core_sku core_client client-service
  12. 12. Extracting Microservices • Step 2: Applications use the service Styling app Warehouse app core_item core_sku core_client client-service
  13. 13. Extracting Microservices • Step 3: Move data to private database Styling app Warehouse app core_item core_sku client-service core_client
  14. 14. Extracting Microservices • Step 4: Rinse and Repeat Styling app Warehouse app core_sku client-service core_client item-service core_item
  15. 15. Extracting Microservices • Step 4: Rinse and Repeat Styling app Warehouse app client-service core_client item-service core_item style-service core_sku
  16. 16. Extracting Microservices • Step 4: Rinse and Repeat Styling app Warehouse app client-service core_client item-service core_item style-service core_sku
  17. 17. Microservice Techniques: Shared Data • Problem o Monolithic database makes it easy to leverage shared data o Where does shared data go in a microservices world? • Principle: Single System of Record o Every piece of data is owned by a single service o That service represents its canonical system of record o Every other copy of that data is a read-only, non-authoritative cache
  18. 18. Microservice Techniques: Shared Data • Approach 1: Synchronous Lookup o Customer service owns customer data o Order service calls customer service in real time order-service customer-service
  19. 19. Microservice Techniques: Shared Data • Approach 2: Async event + local cache o Customer service owns customer data o Customer service sends customer-updated event when customer changes o Order service caches current customer information order-servicecustomer-service
  20. 20. Microservice Techniques: Shared Data • Approach 3: Shared metadata library o Read-only metadata, basically immutable o E.g., size schemas, colors, fabrics, US States, etc. receiving-serviceitem-service style-service
  21. 21. Events as First-Class Construct • “A significant change in state” o Statement that some interesting thing occurred o 0 | 1 | N consumers subscribe to the event, typically asynchronously • Fourth fundamental building block o Presentation  interface / interaction o Application  stateless business logic o Persistence  database o State changes  events • Events represent how the real world works o Finance, software development process, any “workflow”
  22. 22. Microservices and Events • Events are a first-class part of the interface • A service interface includes o Synchronous request-response (REST, gRPC, etc) o Events the service produces o Events the service consumes o Bulk reads and writes (ETL) • The interface includes any mechanism for getting data in or out of the service (!)
  23. 23. Microservice Techniques: Joins • Problem o Monolithic database makes joins very easy o Splitting the data into separate services makes joins very hard
  24. 24. Microservice Techniques: Joins • Approach 1: Service that “Materializes the View” o Listen to events from item-service, events from order-service o Maintain denormalized join of items and orders together in local storage • Best for high cardinality A and high cardinality B (M:N join) Items Orders item-feedback-service item-service order-service
  25. 25. Microservice Techniques: Joins • Many common systems do this o Most NoSQL approaches o “Materialized view” in database systems o Search engines o Analytic systems o Log aggregators
  26. 26. Microservice Techniques: Joins • Approach 2: Join in Application / Client o Get a single customer from customer-service o Query matching orders for that customer from order-service • Best for single A, multiple Bs (1:N join) A B order-history-page customer-service order-service
  27. 27. Microservice Techniques: Joins • Many common systems do this o Web application “mashup”
  28. 28. Microservice Techniques: Workflows and Sagas • Problem o Monolithic database makes transactions across multiple entities easy o Splitting data across services makes transactions very hard
  29. 29. Microservice Techniques: Workflows and Sagas • Transaction  Saga o Model the transaction as a state machine of atomic events • Reimplement as a workflow • Roll back by applying compensating operations in reverse A B C D A B C D
  30. 30. Microservice Techniques: Workflows and Sagas • Many common systems do this o Payment processing flow o Most approval workflows
  31. 31. Microservice Techniques: Workflows and Sagas • Ideal use for Functions as a Service (“Serverless”) o Very lightweight logic o Stateless o Triggered by an event A B C D A B C D ƛ ƛ ƛ ƛ ƛ ƛ ƛ ƛ
  32. 32. Thanks! • @randyshoup • linkedin.com/in/randyshoup

×