Bobby Calderwood at Redis Day NYC 2019

1. PRESENTED BY

2. PRESENTED BY
Hello, I’m Bobby Calderwood!
Software Engineer @Cognitect for 4+ years, Consulting and Datomic
Distinguished Engineer @CapitalOne for 3+ years, helping redesign and move
mission-critical banking systems to the cloud
Founder @Evident_Systems, helping smaller banks redesign and move mission-
critical banking systems to the cloud
Meet with me! http://meet.evident.systems/founder

3. PRESENTED BY
Agenda:
1 Story of a real customer system
Event-Sourcing + Redis + Clojure(Script) = Joy
2 But why that architecture?
A quick primer on CQRS + Event Sourcing using Redis Streams and Clojure(Script)
3 Demo system
Let’s pretend we have to build a system, and then modify it with an asynchronous integration

4. PRESENTED BY
Major automobile manufacturer
Greenfield system for new capability, political proving ground for stack and process
Highly asynchronous image processing and significant UI display/manipulation
Possible need for data re-processing/re-play, undo/redo, audit, and sharing with other
teams
We decided to architect it using Event Sourcing on Streaming Data using Redis,
and implemented in Clojure(Script)
Solving a customer problem

5. PRESENTED BY
But why that architecture?!?

6. PRESENTED BY
At very least, we knew we had to
process images asynchronously
We suspected that we’d need to
keep this pipeline flexible, to try
different processing steps and
techniques
We knew we’d need to convey the
results of this pipeline back to the
UI (the oft-forgotten participant in
the distributed system)
To facilitate our asynchronous processing pipeline

7. PRESENTED BY
Integration among “object/entity-oriented” microservices becomes complex and
burdensome
• Balkanization of data stores
• Data “owner” determines data access pattern
• Non-determinism == impossibility of reasoning about system:
– state
– causality/time
– modes of failure
Because HTTP APIs aren’t enough

9. PRESENTED BY
Request/response protocols face a fundamental epistemological problem
• What is happening, anything new?
• When should I check for new data?
• Who’s responsible to tell me?
Because HTTP APIs aren’t enough

10. PRESENTED BY
Event Sourcing provides “as-of”
consistency across distributed
participants (Online Event Processing)
Streaming Data protocols convey new
data/events to interested participants
(subscribers)
Any participant can simulate synchrony
and consistency by blocking and waiting
for a specific event
Event Sourcing + Streaming Data Facilitate Distributed Consistency
Image by Alan Light CC BY-SA 3.0

11. PRESENTED BY
Cross-cutting concerns need access to your application’s data
• Analytics
• Audit/Compliance/Governance
• Systems Integration
And they don’t want to ETL out of your database…
• Lossy, due to your app’s worldview and assumptions
• Latent
• Requires coordination with you (“we’ll add it to the backlog”)
• Complex
Event Sourcing + Streaming Data Enable Cooperation without Coordination

12. PRESENTED BY
Only one way to cause change to a bounded context
• Send a Command to its API
Two ways to read from a bounded context:
• Listen to its Event stream
– create your own Read Models from one or more Event Streams
– or maybe incorporate subset of Events into your own Event stream
• Ask Queries of its Read Models
CQRS Separates Action from Perception

14. Nasa, Public Domain

15. PRESENTED BY
Why Redis?
(for streaming data)

16. PRESENTED BY
Redis is ubiquitous: you (or your customer) already has it in the stack
Redis is battle-tested, and Ops is comfortable with it
Redis is (often) easier to operate than Kafka (with some tradeoffs)
Redis is Already There, and Just Works

17. PRESENTED BY
After Kafka popularized the stream data structure, Salvatore noticed gap in
Redis, as he described: http://antirez.com/news/114
Existing data structures got almost there
• Sorted Sets: memory hungry, clients can’t block awaiting new records, order
not stable
• Lists: linear time to scan, can’t seek to specific position, no fan-out to multiple
consumers
• Pub/Sub: doesn’t maintain history, can’t do windowed aggregation, etc.
Streams maintain queryable history, and efficiently convey to subscribers
Redis Provides Streams

18. PRESENTED BY
XADD — Produce to a Stream
XRANGE, XREVRANGE — Query stream per range of existing offsets, non-
blocking, batch-ish
XREAD — Optionally-blocking read (conveys novelty), realtime-ish
XREADGROUP — Optionally-blocking read by a group of consumers, ack-ing
messages to advance consumer offset (but watch message ordering!)
But How to Redis Stream?
https://redis.io/topics/streams-intro

19. PRESENTED BY
Why Clojure?

20. PRESENTED BY
Clojure is a fantastic all-purpose programming language whose functional
programming philosophy aligns well with event sourcing, streaming data
We needed to build all components quickly, with a small team of Clojurists:
• UI in ClojureScript
– React + Reagent + re-frame
– Access to JavaScript libraries with Clojure language semantics
• HTTP API in Clojure + Pedestal (+ GraphQL via Lacinia)
• Workers in Clojure
Clojure’s core.async library provide in-process semantics aligned with streaming
data
Clojure(Script) is powerful and full-stack

21. PRESENTED BY
Very solid, simple Redis client
Generated from Redis artifacts
Stays close to Redis semantics
https://github.com/ptaoussanis/carmine/
Clojure has Carmine

22. PRESENTED BY
Let’s Build a Demo System!

23. PRESENTED BY
Store Front Web/Mobile App
How Customers order online, pick up in store
Easy enough, but…
Baristas have to make the drinks
How to coordinate their work?
Should we notify the customers when their order is ready?
Later we discover that managers must monitor inventory levels!
Curiouser and Curiouser!
A Coffee Shop

25. PRESENTED BY
Demo System In Action

26. PRESENTED BY
Build APIs from an immutable log of business events
Using CQRS + Event Sourcing via Redis Streams https://redis.io/topics/streams-intro
Online Event Processing, Martin Kleppmann et. al. https://queue.acm.org/detail.cfm?id=3321612
Try Clojure and ClojureScript https://clojure.org/
Check out the demo app! https://github.com/bobby/redisconf19-demo
Conclusion

27. Thank you!
Thank you!