One of the trickiest problems with microservices is dealing with data as it becomes spread across many different bounded contexts. An event architecture and event-streaming platform like Kafka provide a respite to this problem. Event-first thinking has a plethora of other advantages too, pulling in concepts from event sourcing, stream processing, and domain-driven design.
In this talk, Ben and Cornelia will tackle how to do the following:
● Transform the data monolith to microservices
● Manage bounded contexts for data fields that overlap
● Use event architectures that apply streaming technologies like Kafka to address the challenges of distributed data
Speakers:
Cornelia Davis, Author & VP, Technology, Pivotal
Ben Stopford, Author & Technologist, Office of CTO, Confluent
2. Cornelia
Developer (wasn’t Ops)
Web architectures for >10 years
Cloud-native for 6+ years
Cloud Foundry for 6+ years
Discount code 40% off!: 40cloudnat
https://www.manning.com/books/cloud-native
@cdavisafc
3. Ben Stopford
• Leads Office of the CTO team at
Confluent.
• 20 years in engineering, mostly
distributed data.
• Worked on Kafka’s replication protocol,
rebalancing, throttling.
Download book: bit.ly/benstopford
@benstopford
10. Retries
▸ Client must consider failure
▸ Decide on fall-back
behavior
▸ Likely including retries
▸ But then we need to handle
downstream consequences of
these (retry) behaviors
CLIENT
SERVICE
Timeouts?
If we don’t hear back,
try again
12. Circuit Breakers
▸ Client must consider failure
▸ Decide on fall-back
behavior
▸ Likely including retries
▸ But then we need to handle
downstream consequences of
these (retry) behaviors
CLIENT
SERVICE
CLIENT
CLIENT
CLIENT
13. But then, is stale data better than no data? - often it is
▸ Client must consider failure
▸ Decide on fall-back
behavior
▸ Likely including caches
▸ But then we need to handle
things like cache expiry
SERVICE
APP
SERVICE
APP
APP
SERVICE
APP
SERVICE
APP
SERVICE
APP
15. New LIGHTWEIGHT ARCHITECTURES are emerging Microservices addressing speed to market and cloud scale
Monolithic / Layered Microservices
16.
17. Pattern: Bounded Context
● Domain-Driven Design
● Each bounded context has a single,
unified model
● Relationships between models are
explicitly defined
● A product team usually has a strong
correlation to a bounded context
● Ideal pattern for Data APIs - do not fall
into the trap of simply projecting current
data models
18. Pattern: Database per Service
● Supports Polyglot
persistence
● Independent availability,
backup/restore, access
patterns, etc.
19. Pattern: Database per Service
● Supports Polyglot
persistence
● Independent availability,
backup/restore, access
patterns, etc.
But then what about…
● Overlapping data
● Data ownership
● Data synchronization
22. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
! Response
" Response
# Response
$ Request
%Request
&Request
Connections
API
User:
- id
- username
- name
Connections:
- id
- follower
- followed
Posts
API
Posts:
- id
- date
- userId
- title
- body
Starting with the familiar (Root to Leaf)
23. %
$
!
&
"
#
Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Response
Response
Response
Request
Request
Request
Connections
API
User:
- id
- username
- name
Connections:
- id
- follower
- followed
Posts
API
Posts:
- id
- date
- userId
- title
- body
Layer in caching to improve performance
Cache
How do we handle out of data
data?
25. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
1
2
event store
… …
1
Turn the flow on its head (leaves to root)
1
Connection Created Event - id
- follower
- followed
1
Post Created Event - id
- date
- userId
- title
- body
Cache
(Materialized View)
When the request comes in, the
Connections’ Post service already
has the answer!
26. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
1
1
1
1
2
event store
… …
Cache is populated when things change, not on request
Materialized View
View is formed to match what we need:
“New from your Network”
33. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
event store
… …
Our “Materialized View” is a form of Event Sourcing
View
Cache the view somewhere
convenient
Define the view in our code
34. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
event store
… …
When the applications starts it loads the cache from the log
View
35. Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
event store
… …
Event Store becomes the Source of Truth
Service A
Service E
C
Service D
DB
Apps
Search
NoSQL
Apps Apps
S T R E A M I N G P L A T F O R M
Service C
DB
Apps
Search
NoSQL
Apps App
DWH
Hadoop
S T R E A M I N G P L A T F O R M
C
Service B
DB
Apps
Search
NoSQL
Apps Apps
DWH
Hadoop
S T R E A M I N G P L A T F O R M
37. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
event store
… …
Event Streaming
View
Recommendations
Service
1
2
Recommendation Service plugs
right into the existing flow
38. event store
… …
Event Streaming
Recommendations
Service
KafkaStreamsAPI
Posts
Connections
Page Views by User
Recommendation
Kafka Streams wraps up the .
solutions to these problems in a single API:
- Joins real-time streams
- Tables (from streams)
- Summarises streams
39. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
event store
… …
Event Streaming
View
Recommendations
Service
Create ‘views’ derived from
events in the log
Build event streaming services
using KStreams API
Decouple services from one
another using the event store
47. Join us!
Keynote Speaker:
James Watters
Senior Vice President of Pivotal’s Product
and Business Development
Organizations
Register now at Kafka-Summit.org, and
get a special 25% off using this
code: KSNYPivotal25