Distributed applications are becoming more popular with the increasing popularity of microservices (however you want to define that term). But the principles of distributed application development are key if you want to build a system that is resilient, responsive, elastic and maintainable. In this workshop, we’ll review the principles of CQRS and the Reactive Manifesto, and how they complement each other. We’ll build an application that can handle a large stream of data, and allow users to still have a responsive experience while interacting with real-time and near-real-time data. We’ll look at Akka.NET as the workhorse inside your services, and how the principles of CQRS can help with your service-to-service communications. We’ll also look at how Event Sourcing can aid in managing your domain state, and how an event stream can be used to project data for your system for a number of different uses. We’ll build our own simple event store, but also look at commercially available stores, too. This session will focus on using Akka.NET along with a few other tools and technologies, such as EventStore and MongoDB. The concepts learned in this session will be applicable to a number of different tools, technologies and languages.

1. Reactive Development
Commands, Actors and Events – Oh My!

2. About Me
• 6-Time .NET (Visual C#) MVP (April 2011)
• Sr. Solutions Architect at Confluence
• One of the organizers for Pittsburgh TechFest
(http://pghtechfest.com)
• Organizer of Pittsburgh Reactive Dev Group
(http://meetup.com/reactive)
• Past President of Pittsburgh .NET Users Group
• Twitter - @DavidHoerster
• Blog – http://blog.agileways.com

3. Agenda
• What are we building?
• Messages
• What is CQRS?
• …and isn’t it dead?
• Build a simple handler
• Using Actors Instead of Procedural Domains
• Communicating Back to the Client
• Resilient Processing

4. Schedule
Start End Topic
8:30 AM 8:45 AM Intro / What are we building
8:45 AM 10:00 AM Part 1: CQRS / Messages / Lab 1 (Building a Handler)
10:00 AM 10:15 AM Break
10:15 AM 11:45 AM Part 2: Actors / Mini Labs
11:45 AM 12:30 PM Lunch!!
12:30 PM 2:00 PM Part 3: CQRS + Actors / Lab 2 (Actors in Handler; Projections)
2:00 PM 2:15 PM Break
2:15 PM 3:00 PM Lab 3 – End to End Communication (Real-Time Updates)
3:00 PM 3:45 PM Lab 4 – Events; Lab 5 – Querying Events (Restoring Actors from Events;
Querying EventStore)
3:45 PM 4:00 PM Wrap-Up

5. What Are We Building?
• An application that processes multiple files
• Each file contains all of the plays from the 2015 baseball season for one
HOME team
• The files are CSV
• Each row of the file will be sent as a message
• A back end process will subscribe to the message and process the data
• Game data, batter data, pitcher data
• Results will be published to a topic
• Another service will subscribe to that message and push updates to a
simple HTML page via SignalR

6. Baseball Play Processor System
Producer
Producer
Producer
Topic
Consumer
Topic
API
Gateway
Consumer
Projections
(Mongo)
Events
(EventStore)
Ad Hoc
Query
Retro
Files

7. Retro Files
• RetroSheet.org files listing every play for every home game in 2015
(per team)
• CSV files
• 97 fields – we only use a handful
• http://www.retrosheet.org/datause.txt
• Feel free to use more fields!!!

8. Producer and Consumer
• Separate processes
• Producer reads in Retro CSV files
• Pushes each row into a message “topic”
• Consumer subscribes to the new messages on the topic
• Handles the row accordingly
• Consumer will eventually use actors to process data
• Run multiple producers, consume additional fields, create new actors,
create new projections!!!

9. Message “Topics”
• Using a RethinkDB table per topic
• Somewhat of a workaround for local development
• EXE is a beta preview for Windows, so not ready for prime time
• RethinkDB allows clients to subscribe to changes
• Acts like a topic
• Multiple subscribers
• C# client is a port of the official Java client

10. MongoDB
• Document database
• Holds projections for batters and pitchers
• And any other projections you’d like to create 
• Using 3.X version with WiredTiger
• RoboMongo 0.9.X supports WiredTiger (finally!)
• MongoDB C# client is official client

11. API Gateway
• OWIN self-hosted Web API service
• Subscribes to changes for batters
• Exposes simple API for retrieving batter projection from MongoDB
• Pushes changes from topic to client via SignalR
• Only pushes to subscribed “group” – not broadcasting out all changes
• Create more pages and APIs!!!

12. Web Client
• Stupid simple web site
• One static HTML page
• Uses jQuery for simple interaction
• Uses SignalR JS library to subscribe to hub for batter changes
• As new batter is queried, that is the “group” that is subscribed to
• Feel free to create new pages, enhance look n’ feel, add more data!!!

13. EventStore
• Used to hold events for players
• Each player id is a new stream
• Stream consists of 0 to many events
• Can be of different types
• Events are used to restore state of player actors (batters for now) and
also used for ad hoc querying later
• Feel free to add new types of events (pitcher events???) and add
more data!!!

14. Ad Hoc Querying
• Queries the EventStore for basic batter information
• Average and Homers hit for each count (balls/strikes)
• Some overall stats
• Uses EventStore C# client (official)
• Create new queries, make it a web page, try to slice and dice data
differently!!!

15. Perspectives on Procedural

16. Real World Scenario
File File File
Join
Op
Join/
End

17. Real World Scenario
• Actually looks like this:
• Multiple sources
• Several transforms
• Several joins
• Configurable

18. Messages as Part of Application
Design

19. CQRS
• Command Query Responsibility Segregation
• Coined by Greg Young
• Evolution of CQS (Command Query Separation)
• Coined by Bertrand Meyer
• Both revolve around the separation of writes (command) and reads
(query)
• CQS is more at the unit level
• CQRS is at the bounded context level, or so

20. Messages
• The core of CQRS  Messages
• Communication from the service layer to the domain via commands
• Command handlers could also update the underlying repository
• As actions happen, events are raised
• Event handlers could issue other commands
• Event handlers could update the underlying repository

21. Messages
• Regardless of how it’s implemented, communication between
application parts is via messages
• Not only helps communicate intent of the action
• “PublishWidget”, “ExpireWorkItem”, “UpdateDefinition”, “DefinitionUpdated”
• But allows for remote handling of messages
• Queue, REST payload, etc.

22. Messages are key to being reactive

23. So?
• So how does this affect my system design?
• Let’s consider some procedural pitfalls

24. Word Counter Example
• Simple program to count the occurrences of words in a document
• Print out the top 25 words and their counts at the end
• Can be accomplished easily via procedural code

25. Word Counter Example

26. Word Counter Example
• What if you wanted to spread out the counting load?
• What if you wanted to scale out to other nodes/processes/machines?
• Fault tolerance, concurrent actions, etc.?

27. Building Concurrent Apps
• In word counter, may need to spawn some threads
• Lock some critical sections, etc.
• The system becomes responsible for knowing everything going on
• Very Proactive (not Reactive!) system

28. Side note: reactive vs. proactive
• Isn’t proactive a good thing?
• A system taking upon itself to
check state is proactive
• It’s doing too much
• A reactive system reacts to
events that occur
• It’s doing just the right amount

29. Side note: reactive vs. proactive
• Isn’t proactive a good thing?
• A system taking upon itself to
check state is proactive
• It’s doing too much
• A reactive system reacts to
events that occur
• It’s doing just the right amount
System
EvtEvt
System
EvtEvt
Proactive
Reactive

30. What is CQRS?

31. CQRS Background
• Command Query Responsibility Segregation
• Coined by Greg Young
• Heavily influenced by Domain Driven Design (DDD)
• Evolution of Meyer’s CQS (Command Query Separation)
• Separation of writes (command) and reads (query)
• CQS is more at the unit level
• CQRS is at a higher level  bounded context / service

32. What is CQRS?
• Simply: separate paths for reads and writes
• Communicate via messages
• Commands and Events
• “CreateOrder” and “OrderCreated”
• Reads are against a thin read model (preferably optimized)

33. CQRS Visualized: Basic
Client
Backend
Commands Queries

34. CQRS
Visualized
Controller
Handler
Domain
Persistence /
Read Model
Read LayerCommand
Side
Query
Side

35. CQRS Extended
• Can move from direct references to queued
• Decouples handlers
• Increased isolation of areas
• Leads to single message causing several changes
• Raising of events and subscriptions to events

36. CQRS
Visualized
Controller
Handler
Domain
Persistence
Read Layer
Handler
Command/EventTopics
Read Model
Command
Side
Query
Side

37. CQRS Handler

38. CQRS Handler

39. Benefits
• Message Based
• Segregation of Responsibilities – Separate Paths
• Smaller, simpler classes
• Albeit more classes
• Focused Approach to Domain
• Move away from anemic domain models

40. CQRS Principles
• Separate Paths for Command and Queries
• Message-Based
• Async Friendly
• Thin read layer
• Don’t fear lots of small classes

41. Is CQRS Dead?

42. <rant>

43. "CQRS is Hard"
• Misunderstandings
• Has to be async
• Has to have queueing
• Need a separate read model
• Need to have Event Sourcing
• Perceived as overly prescriptive, but little prescription
• Confusing
• Command Handler vs. Event Handler vs. Domain
• Command vs. Event

44. Has CQRS “Failed”?
• Focus on CQRS as architecture over
principles
• Search for prescriptive guidance
• Infrastructure/Implementation debates
• Should domains have getters??
• “…there are times a getter can be
pragmatic. The trick is learning where to use
them.” – Greg Young (DDD/CQRS Group)
• “This list has its heads up its own
[butts] about infrastructure. What
business problems are you working
on?” -- Greg Young (DDD/CQRS Group)

45. </rant>

46. Popular Architecture Topics
• Microservices
• Event driven
• Distributed computing
• NoSQL
• Async

47. But Isn’t CQRS…?
Message Driven
Responsive Read Model
Isolated Handlers
Async Friendly
Group Handlers into Clusterable Groups
NoSQL Friendly (Flattened Read Models)

48. CQRS Can Help Drive Reactive Apps

49. Lab: Building a Simple Handler

50. Lab Info
• Scaffold: Lab1-Handler
• Create a Producer EXE (service) that adds Plays to a RethinkDB table
• Create a Consumer EXE (service) that subscribes to the “plays” RethinkDB table for changes
• Use the Shared project for common entities and messages
• In RethinkDB, make sure you:
• Create a database named “baseball” and a table named “plays”
• To clear the table, go to Data Explorer and issue this command:
r.db("baseball").table("plays").delete();

51. Lab: Using RethinkDB
• What is and Why
RethinkDB?
• Document database
• Can join tables/collections
• Big feature is subscribing
to changes
• Inserting into a table is
just adding a POCO

52. Lab: Using RethinkDB for Subscriptions
• RunChanges is
blocking
• Use Reactive
Extensions to
subscribe to
changes in the feed
• “include_initial” is
an OptArg to
indicate you want
all rows to start
with along with
changes.

53. CQRS Evolved – CQRS + Actors

54. CQRS & the Reactive Manifesto
• Message Based
• Core of CQRS
• Responsive
• Commands are ack/nack
• Queries are (can be) against an optimized read model
• Resillient
• Not core to CQRS  Implementation
• Elastic
• Not core to CQRS  Implementation

55. CQRS + Reactive
• Resillient and Elastic core to
Reactive
• CQRS’ core is Message-Based
and Responsive
• Combining the two is powerful

56. Actor Model
CQRS lacks prescriptive guidance
• Actor Model provides a message-based pattern
• Provides prescriptive guidance
• Makes for more generalized implementation
• Actor Model is driving more reactive-based development

57. Actor Model
• Actor Model is a system made of small units of concurrent
computation
• Formulated in the early 70’s
• Each unit is an actor
• Communicates to each other via messages
• Actors can have children, which they can supervise

58. What is an Actor?
• Lightweight class that encapsulates state and behavior
• State can be persisted (Akka.Persistence)
• Behavior can be switched (Become/Unbecome)
• Has a mailbox to receive messages
• Ordered delivery by default
• Queue
• Can create child actors
• Has a supervisory strategy for child actors
• How to handle misbehaving children
• Are always thread safe
• Have a lifecycle
• Started, stopped, restarted

59. Actor Hierarchy
baseball
gameCoord
inator
gameInfo-x gameInfo-y
playerSupe
rvisor
batter-abatter-bbatter-c
c-01 c-11 c-32

60. Akka: Resiliency
• Actors supervise their children
• When they misbehave, they are notified
• Can punish (fail) all children, or tell the misbehaving on to try again
• Protects rest of the system

61. Akka.NET
• Akka.NET is an open source framework
• Actor Model for .NET
• Port of Akka (for JVM/Scala)
• http://getakka.net
• NuGet – searching for “akka.net”, shows up low in the list

62. Starting with Akka.NET
• Akka.NET is a hierarchical system
• Root of the system is ActorSystem
• Expensive to instantiate – create and hold!
• ActorSystem can then be used to create Actors

63. Creating an Actor
• An Actor has a unique address
• Can be accessed/communicated to like a URI
• Call ActorOf off ActorSystem, provide it a name
• URI (actor path) is created hierarchically
• Actor Path = akka://myName/user/announcer

64. Creating Children
• Best to call ActorOf to get an IActorRef
• Create a static Props creator
• Actor path is relative to parent’s actor path
• akka://myDemo/user/countChocula/{childName}
• When creating a child, good idea to name it
• So you can reference it later!

65. Sending a Message
• Messages are basis of actor communication
• Should be IMMUTABLE!!!
• “Tell” an actor a message
• Async call
 Immutable!!

66. Receiving A Message
• Create your Actor
• Derive from ReceiveActor
• In constructor, register your
message subscriptions
• Looks similar to a CQRS
• Handler

67. Receiving A Message
• Each actor has a mailbox
• Messages are received in the actor’s mailbox (queue)
• Actor is notified that there’s a message
• It receives the message
• Takes it out of the mailbox
• Next one is queued up
• Ordered delivery by default
• Other mailboxes (like Priority) that are out-of-order
Actor
Mailbox
Msg

68. Async
• Actors have messages delivered to them in order via a Mailbox
• As message is received, handed to Actor to process
• Messages in mailbox queue up
• As a result, async processing of messages isn’t possible
• Receive<> (async msg => { … }) doesn’t work
• If you need to perform async tasks within Receive
• PipeTo(Self) usually does the trick
• Re-queues message back on Mailbox
• When delivered, actor resumes
Actor
Mailbox
Msg
PipeTo

69. Lab: Breaking Bad Actors
And other simple actor projects

70. Lab Info
• Scaffold: A1-HelloAkka
• Simple Two Actor Project
• Try including another actor or two (and another message!)

71. Lab Info
• Scaffold: A2-ActorTimeoutTest
• Show how an actor can commit “suicide”
• Uses a Scheduler
• Try to implement this another way
• Instead of parent issuing the Poison Pill, have the Child do it
• Override AroundReceive()

72. CQRS + Actors

73. Elements of CQRS in Actor Model
• Message-based
• Potentially async
• Focused code
• Actor System is your Command Side

74. CQRS + Actors
• Combining the principles of CQRS and the patterns of the Actor
Model
• Message-based communication with concurrent processing

75. CQRS + Actors: Messages
• Messages in an Actor System include both Commands and Events
• Still encapsulate intent
• Still should still contain information necessary to process/handle
command or event
• Basically, no big change here

76. CQRS + Actors: Messages

77. Lab: Actor Word Counter

78. Lab Info
• Scaffold: A3-ActorWordCounter
• Notice how actors can call each other (and themselves)
• Update messages to be more like Commands and Events
• CHALLENGE: Can you use the RoundRobinPool???

79. CQRS + Actors – Handlers and
Services

80. CQRS + Actors: Handlers
• Handlers begin to encapsulate your
Actor System
• Generally a single Actor System per
handler
• Have several actors at top level of
hierarchy to handle initial
messages
• Domain can be injected into the
Actor System via Create()
Handler
Client
Queue
Persistence
Other Handler
Direct call or via queue

81. CQRS + Actors: Handlers

82. CQRS + Actors: Handlers

83. CQRS + Actors: Handlers
• Handlers act as the entry point
• Web API
• Service subscribed to queue
• CQRS  IHandle<T> for those messages “exposed”
• Actor System is called by the CQRS Handler
• Actor System has many other “internal” actors

84. CQRS + Actors: Services
Word Counter Handler
super
Count Write Stats
A B G H Y Z

85. Word Counter
Handler
CQRS + Actors: Services
super
Count
A B
Word Writer Handler
super
Write
G H
Word Stats Handler
super
Stats
Y Z
Message Bus

86. CQRS + Actors + Services
Service A Service B Service C
Message Bus
DW / DL /
Read Model
Cmds / Events Cmds / Events
API Gateway SvcCommands Queries
RM RMRM
Queries (?)

87. Service A
Service A
Service A
CQRS + Actors + Services + Cluster
(Akka.Cluster)
Service A Service B Service C
Message Bus
DW / DL /
Read Model
Cmds / Events Cmds / Events
API Gateway SvcCommands Queries
RM RMRM
Queries (?)

88. CQRS + Actors + Services
• Actors act as the workers for your handlers
• Services encapsulate handler into a bounded context
• CQRS is the messaging and communication pattern within system
• Using these three helps build killer reactive applications

89. Lab: Distributed Reactive Apps

90. Lab Info
• Scaffold: Lab2-GameActor
• The DomainActors and Messages folders have been excluded from
the project
• Include them if you want to run things
• Study them if you want to write your own

91. Lab Info
• RethinkDB Setup
• Database named: “baseball”
• Tables named: “plays” and “batterStat”
• To clear these tables, issue the command from Data Explorer
r.db("baseball").table("batterStat").delete();
r.db("baseball").table("plays").delete();
• MongoDB shouldn’t need setup – database and collections should be
created
• Database is “baseball2015” and collections are “batter” and “pitcher”

92. Lab Info
• Add a new collection to MongoDB’s baseball2015 database via
mongoimport
• Command line
• Binmongoimport /d baseball2015 /c players /headerline /type csv
“dataplayers.csv”
• This will allow you to view players’ real names instead of just IDs (e.g.
Andrew McCutchen vs. mccua001)

93. Lab Info: Actor Model
/user
gameEventCoord
gameSupervisor
gameActor 1 gameActor 2
teamSupervisor
teamActor 1 teamActor 2
batterSupervisor
pitcherSupervisor
pitcherActor 1 pitcherActor 2
batterActor 1 batterActor 2

94. Lab: Communicating Back

95. Lab Info
• Scaffold: Lab3-Display
• This lab subscribes to a topic/table in RethinkDB for batter events
• Events are pushed to a web JavaScript client via SignalR
• Web API uses SignalR groups to only send subscribed players to clients
(instead of all updates)
• DomainActors and Messages are excluded. Use if you’d like
• UI-ApiHubs and UI-ApiUtils are excluded. These set up SignalR Hubs and
Group Subscriptions. Use if you’d like.

96. Lab: End to End

97. Lab Info
• Scaffold: Lab4-Events
• Nothing excluded
• Try to add additional information to events or add pitcher events to
EventStore
• Add a pitcher.html page, similar to default.html, to query on pitcher
information

98. Lab Info
• Scaffold: Lab5-EventStoreSample
• Use a populated EventStore to query
• Slice events differently
• Create different types of queries
• Also, try to incorporate this into a web page instead of a console app
• Show Batter Count stats on a web page, even in real time via SignalR
updates!!!

99. Thank you!