Managing Data at Scale - Microservices and Events

Managing Data at Scale
Microservices and Events
Randy Shoup
@randyshoup
linkedin.com/in/randyshoup

Evolution to
Microservices
• eBay
• 5th generation today
• Monolithic Perl  Monolithic C++  Java  microservices
• Twitter
• 3rd generation today
• Monolithic Rails  JS / Rails / Scala  microservices
• Amazon
• Nth generation today
• Monolithic Perl / C++  Java / Scala  microservices
@randyshoup linkedin.com/in/randyshoup

No one starts with microservices
…
Past a certain scale, everyone
ends up with microservices

First Law of Distributed Object
Design:
Don’t distribute your objects!
-- Martin Fowler

Managing Data
at Scale
• Migrating to Microservices
• Challenges of Data in Microservices
• Challenges of Event-Driven Systems

Microservices
• Single-purpose
• Simple, well-defined interface
• Modular and independent
• Isolated persistence (!)
A
C D E
B

Extracting
Microservices
• Problem: Monolithic shared DB
• Clients
• Shipments
• Items
• Styles, SKUs
• Warehouses
• etc.
stitchfix.com Styling app Warehouse app Merch app
CS app Logistics app Payments service Profile service

Extracting
Microservices
• Decouple applications / services from shared DB
• Clients
• Shipments
• Items
• Styles, SKUs
• Warehouses
• etc.
stitchfix.com Styling app Warehouse app Merch app
CS app Logistics app Payments service Profile service

Extracting
Microservices
• Decouple applications / services from shared DB
Styling app Warehouse app
core_item
core_sku
core_client

Extracting
Microservices
• Step 1: Create a service
core_item
core_sku
core_client
client-service

Extracting
Microservices
• Step 2: Applications use the service
core_item
core_sku
core_client
client-service

Extracting
Microservices
• Step 3: Move data to private database
core_item
core_sku
client-service
core_client

Extracting
Microservices
• Step 4: Rinse and Repeat
core_sku
client-service
core_client
item-service
core_item

Extracting
Microservices
• Step 4: Rinse and Repeat
client-service
core_client
item-service
core_item
style-service
core_sku

Managing Data
at Scale
o Two Architectural Tools
o Shared Data
o Joins
o Transactions

Service as
System of Record
• Single System of Record
o Every piece of data is owned by a single service
o That service is the canonical system of record for that data
• Every other copy is a read-only, non-authoritative
cache
customer-service
styling-service
customer-search
billing-service

Events as
First-Class Construct
• “A significant change in state”
o Statement that some interesting thing occurred
• Traditional 3-tier system
o Presentation  interface / interaction
o Application  stateless business logic
o Persistence  database
• Fourth fundamental building block
o State changes  events
o 0 | 1 | N consumers subscribe to the event, typically asynchronously

Microservices
and Events
• Events are a first-class part of a service interface
• A service interface includes
o Synchronous request-response (REST, gRPC, etc)
o Events the service produces
o Events the service consumes
o Bulk reads and writes (ETL)
• The interface includes any mechanism for getting data in
or out of the service (!)

Data in Microservices:
Shared Data
• Monolithic database makes it easy to leverage shared
data
• Where does shared data go in a microservices world?

Shared Data
Option 1: Synchronous Lookup
o Customer service owns customer data
o Fulfillment service calls customer service in real time
fulfillment-service
customer-service

Shared Data
Option 2: Async event + local cache
o Customer service owns customer data
o Customer service sends address-updated event when customer address
changes
o Fulfillment service caches current customer address
fulfillment-servicecustomer-service

Shared Data
Option 3: Shared metadata library
o Read-only metadata, basically immutable
o E.g., size schemas, colors, fabrics, US States, etc.
receiving-serviceitem-service
style-service

Joins
• Monolithic database makes it easy to join tables
• Splitting the data across microservices makes joins very
hard
SELECT FROM A INNER JOIN B ON …

Joins
Option 1: Join in Client Application
o Get a single customer from customer-service
o Query matching orders for that customer from order-service
Customers
Orders
order-history-page
customer-service order-service

Joins
Option 2: Service that “Materializes the View”
o Listen to events from item-service, events from order-service
o Maintain denormalized join of items and orders together in local storage
Items Order Feedback
item-feedback-service
item-service
order-feedback-service

Joins
• Many common systems do this
o “Materialized view” in database systems
o Most NoSQL systems
o Search engines
o Analytic systems

Transactions
• Monolithic database makes transactions across multiple
entities easy
• Splitting data across services makes transactions very
hard
BEGIN; INSERT INTO A …; UPDATE B...; COMMIT;

“In general, application
developers simply do not
implement large scalable
applications assuming
distributed transactions.”
-- Pat Helland
Life After Distributed Transactions: An Apostate’s Opinion, 2007

“Grownups don’t use
distributed transactions”
-- Pat Helland

Workflows and Sagas
• Transaction  Saga
o Model the transaction as a state machine of atomic events
• Reimplement as a workflow
• Roll back by applying compensating operations in
reverse
A B C
A B C

Workflows and Sagas
• Many common systems do this
o Payment processing
o Expense approval
o Travel
o Any multi-step workflow

Workflows and Sagas
• Simple event-driven processing
o Very lightweight logic
o Stateless
o Triggered by an event
•  Consider Function-as-a-Service (“Serverless”)
A B C
A B C
ƛ ƛ ƛ
ƛ ƛ ƛ

Managing Data
at Scale
o Event Duplication
o Event Ordering

Event
Duplication
• Problem: The same event will be delivered more than
once
o Network issues
o Redelivery
Producer ConsumerTransport
• Event-1
• Event-2
• Event-3
• Event-1
• Event-2
• Event-2
• Event-3

Event
Duplication
• The consumer must process an event correctly
regardless of how many times it receives it
• Event-1
• Event-2
• Event-3
• Event-1
• Event-2
• Event-2
• Event-3
• Event-1
• Event-2
• Event-3

Event Duplication:
(A) Exactly Once Delivery
Message bus buffers messages
o Message bus remembers events it has delivered, identified by message id
o Only deliver event if not yet delivered
• Event-1
• Event-2
• Event-3
• Event-1 [1]
• Event-2 [2]
• Event-3 [3]
• Event-1 [1]
• Event-2 [2]
• Event-2 [2]
• Event-3 [3]
• Event-1 [1]
• Event-2 [2]
• Event-3 [3]

Event Duplication:
(B) Idempotent Processing
Option 1: Idempotency key
o Remember previously processed events, identified by idempotency key
o Before processing, check whether you have processed it already
o E.g., counter
• Event-1 [aaa]
• Event-2 [bbb]
• Event-3 [ccc]
• Event-1 [aaa]
• Event-2 [bbb]
• Event-2 [bbb]
• Event-3 [ccc]
• Event-1
• Event-2
• <nothing>
• Event-3
aaa
aaa,bbb
aaa,bbb
aaa,bbb,ccc

Event Duplication:
Option 2: Processing is inherently idempotent
o Simply do the processing N times for N events
o E.g., “set X to 10”, UPSERT
• Event-1
• Event-2
• Event-3
• Event-1
• Event-2
• Event-2
• Event-3
• Set x:=1
• Set x:=2
• Set x:=2
• Set x:=3

Event Duplication:
Option 3: Conflict-free Replicated Datatypes (CRDTs)
o Achieve agreement without explicit coordination
o Custom data structures, composable processing steps
o Many implementations, but still an area of active research
Common techniques
o Remember what you saw (request id, idempotency key)
o Remember that an item was deleted (“tombstone”)
(Do *NOT* roll your own from first principles)

Event
Ordering
• Problem: Events will arrive out of order
o Network issues
o Processing time
o Redelivery
• Event-1
• Event-2
• Event-3
• Event-2
• Event-1
• Event-3

Event
Ordering
• The consumer must process events correctly regardless
of the order in which they arrive
• Event-1
• Event-2
• Event-3
• Event-2
• Event-1
• Event-3
• Event-1
• Event-2
• Event-3

Event Ordering:
(A) Impose Order
Option 1: Sequence + Reorder in the message bus
o Sequence number at the producer
o Message bus queues messages, waits for gaps
o Bus sends to consumer in order
• Event-1 [1]
• Event-2 [2]
• Event-3 [3]
• Event-1 [1]
• Event-2 [2]
• Event-3 [3]
• Event-1 [1]
• Event-2 [2]
• Event-3 [3]
• Event-2 [2]
• Event-1 [1]
• Event-3 [3]

Event Ordering:
(A) Impose Order
Option 2: Sequence + Reorder in the consumer
o Sequence number / timestamp at the producer
o Consumer reorders before processing
• Event-1 [1]
• Event-2 [2]
• Event-3 [3]
• Event-2 [2]
• Event-1 [1]
• Event-3 [3]
• Event-1 [1]
• Event-2 [2]
• Event-3 [3]

Event Ordering:
(B) Order-Independence
Option 1: Order-independent semantics
o E.g., count number of events
• Event-1
• Event-2
• Event-3
• Event-2
• Event-1
• Event-3
• +1
• +1
• +1

Event Ordering:
Option 2: Notification + Read-back
o Event is a notification: object id + type of change
o Consumer “reads back” to the source service to get current state
• Event-1
• Event-2
• Event-3
• Event-2
• Event-1
• Event-3
• State = 2
• State = 2
• State = 3

Event Ordering:
Option 3: Event Sourcing
o Store all events in a log
o Process / interpret later
• Event-1
• Event-2
• Event-3
• Event-2
• Event-1
• Event-3
Event-2
Event-1
Event-3
• Event-1
• Event-2
• Event-3

¡Gracias!
@randyshoup
linkedin.com/in/randyshoup
medium.com/@randyshoup

Managing Data at Scale - Microservices and Events

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Managing Data at Scale - Microservices and Events

Similar to Managing Data at Scale - Microservices and Events (20)

More from Randy Shoup

More from Randy Shoup (19)

Recently uploaded

Recently uploaded (20)

Managing Data at Scale - Microservices and Events