Real World Akka Actor Recipes JavaOne 2013

Real -World Akka Actor Recipes
Björn Antonsson
@bantonsson
co-authored with
Jamie Allen and Patrik Nordwall
Tuesday, 1 October 13

@bantonsson@bantonsson
Overview
• What is Akka?
• What are actors?
• Getting into the flow
• Getting your message across
• Tying it all together
2

What is Akka?

What is Akka?
• Toolkit and runtime for reactive applications
• Write applications that are
– Concurrent
– Distributed
– Fault tolerant
– Event-driven
4

What is Akka?
• Has multiple tools
– Actors
– Futures
– Dataflow
– Remoting
– Clustering
5

What are actors?

What are actors?
• Isolated lightweight event-based processes
• Share nothing
• Communicate through async messages
• Each actor has a mailbox (message queue)
• Location transparent (distributable)
• Supervision-based failure management
7

What is an actor good for?
• An island of sanity in a sea of concurrency
• Everything inside the actor is sequential
– Processes one message at a time
• Very lightweight
– Create millions
– Create short lived
• Inherently concurrent
8

Actors compared to Objects
• Think of an Actor as an Object
• You can't peek inside it
• You don't call methods
– You send messages (asynchronously)
• You don't get return values
– You receive messages (asynchronously)
• The internal state is thread safe
9

Why should I care?

The world is multicore!

12
Amdahl’s Law

So what's the catch?
• Really no catch
• A different programming paradigm
• All about tradeoffs
– Some things are easier some harder
• Think different
13

Getting into the flow

Getting into the flow
• Why do you need flow control?
• How do you control the flow?
• Do you really need all messages?
15

Why do you need flow control?
• Function calls are blocking
• Message sends are asynchronous
• Possible problems
– Produce jobs too fast
– Many jobs need much CPU and/or Memory
– External resources have limits
– Unpredictable job patterns
16

Why do you need flow control?
• Free flow of messages can lead to
– Blocked from external resource
– Actor mailbox backup
– Slow system
– Out of Memory
17

How do you control the flow?
• Push with rate limiting
– A fixed number of jobs per time unit
• Push with acknowledgment
– A fixed number of jobs can be in progress
– New jobs are started a"er old jobs finish
• Pull
– New jobs are pulled as old are completed
18

Push with rate limiting
• A timer sends ticks at fixed intervals
• On every tick the master gets new tokens
• When there are no tokens, jobs get queued
• When there are tokens, start queued jobs
19

20
Worker
Master [0]
Queue [0]
Timer
Worker
1. Work

21
Worker
Master [0]
Queue [1]
Timer
Worker
1. Work 2. Enqueue

22
Worker
Master [5]
Queue [1]
Timer
Worker
1. Work 2. Enqueue
3. Tick

23
Worker
Master [4]
Queue [0]
Timer
Worker
1. Work 2. Enqueue
4. Dequeue
3. Tick

24
Worker
Master [4]
Queue [0]
Timer
Worker
1. Work 2. Enqueue
4. Dequeue
3. Tick 5. Work

Push with acknowledgement
• A fixed number of jobs are started
• Wait for ACK before starting more jobs
• Jobs that can't be started are queued
• When ACK arrives start queued job
• To keep workers busy, send more than one
job per worker
– Use a high water mark to stop sending and a low
water mark to start sending
25

26
Worker
Master [0]
Queue [0]
Worker
1. Work

27
Worker
Master [0]
Queue [1]
Worker
1. Work 2. Enqueue

28
Worker
Master [1]
Queue [1]
Worker
1. Work 2. Enqueue
3. Ack

29
Worker
Master [0]
Queue [0]
Worker
1. Work 2. Enqueue
4. Dequeue
3. Ack

30
Worker
Master [0]
Queue [0]
Worker
1. Work 2. Enqueue
4. Dequeue
5. Work
3. Ack

31
Worker
Master [1]
Queue [0]
Worker
1. Work 2. Enqueue
4. Dequeue
5. Work
6. Ack
3. Ack

Pull
• Incoming jobs are queued
• Workers ask for jobs
• Jobs are handed out when available
• Workers don't do active polling
• Can lead to lag if jobs are small compared to
cost of getting a new job
– Use batching to counteract lag
32

33
Worker
Master
Queue [0]
Worker
1. Work

34
Worker
Master
Queue [1]
Worker
1. Work 2. Enqueue

35
Worker
Master
Queue [1]
Worker
1. Work 2. Enqueue
3. Work?

36
Worker
Master
Queue [0]
Worker
1. Work 2. Enqueue
4. Dequeue
3. Work?

37
Worker
Master
Queue [0]
Worker
1. Work 2. Enqueue
4. Dequeue
5. Work
3. Work?

Do you really need all messages?
• Group messages together
– Batching
• Discard/Aggregate messages
– Scrubbing
38

Batching
• Collect a number of messages before
sending/processing them
– A predefined number of messages or time
• Useful for things like
– Write behind
– Database bulk insert/update
– Heavyweight operations e.g. GUI rendering
39

Scrubbing
• Discard or aggregate some messages
– Predefined number of messages or time
• Useful for things like
– Financial market data
– Statistics
40

Getting your message across

Getting your message across
• When is a message delivered?
• The fallacy of guaranteed delivery
• What Akka guarantees
• Reliable messaging
42

When is a message delivered?
• Function calls block until done
• Message sends return immediately
• Which is the right point?
– Sent/Received Network?
– Enqueued/Dequeued Mailbox?
– Processed by Actor?
• Do ACKing at the business level
43

Guaranteed delivery
• From Enterprise Integration Patterns
• Messaging system uses built-in store to
persist
• ACK everywhere
– Producer to sender
– Sender to receiver
– Receiver to consumer
44

Lots of ACKs. What if I just...
• Use Durable Mailboxes?
– When is the message in the mailbox?
– No guarantees that it ever got there
– Still have to ACK to be certain
45

Lots of ACKs. What if I just...
• Use an External Durable Message Queue
– A SPOF/Bottleneck?
– When is the message in the message queue?
– The queue does ACKing internally
– No guarantees that it ever gets out
– Still have to ACK to be certain
46

Guaranteed delivery doesn't exist
• Things break
– Persistent store crashes
– Network fails
– Server goes down
• Design for failure and resilience
• Do ACKing at the business level
47

What Akka guarantees
• At most once delivery
– Message is only delivered once, if at all
– The weakest guarantee
• Ordered per actor sender-receiver pair
– Actor A sends messages to actor B
– If the messages are received by actor B,
it will be in the order as sent by actor A
48

Other delivery guarantees
• At least once
– Message will eventually be delivered
– Can happen multiple times
• Exactly once
– Message will eventually be delivered
– Will only happen once
• Have to add these yourself
– They involve ACKing ;)
49

Reliable Messaging: At least once
• Send with acknowledge
– Keep sending until you get an ACK
• Receive with re-request
– When missing a message request it
– Needs unique sequence numbers
• Requires
– Message store at sender (available/redundant)
50

51
Sender
Receiver
Store

52
Sender
Receiver
Store
1. Message

53
Sender
Receiver
Store
2. Message

54
Sender
Receiver3. Ack
Store
2. Message

55
Sender
Receiver3. Ack
Store
4. Ack
2. Message

Reliable Messaging: At least once
• Use a pipe of actors before ACKing
• Keep pipe free of side eﬀects
– Same message might come several times
• ACK should be done at the business level
56

57
Sender
Check
Inventory
Authenticate
ReceiverStore

58
Sender
Check
Inventory
Authenticate
ReceiverStore
1. Message

59
Sender
Check
Inventory
Authenticate
Receiver
2. Message
Store

60
Sender
Check
Inventory
Authenticate
Receiver
2. Message
3. Message'
Store

61
Sender
Check
Inventory
Authenticate
Receiver
4. Message''
2. Message
3. Message'
Store

62
Sender
Check
Inventory
Authenticate
Receiver
4. Message''
2. Message
3. Message'
5. Ack
Store

63
Sender
Check
Inventory
Authenticate
Receiver
4. Message''
2. Message
3. Message'
5. Ack
Store
6. Ack

Receive with re-request
• When missing a message request it
• Requires
– Uniquely identifiable sequence
– Message store at sender (available/redundant)
64

65
Sender
Receiver
Store

66
Sender
Receiver
Store
1. Message

67
Sender
Receiver
Store
2. Message

68
Sender
Receiver
Store
3. Message'

69
Sender
Receiver
Store
4. Message'

70
Sender
Receiver5. Request
Store
4. Message'

71
Sender
Receiver
Store
6. Message

72
Sender
Receiver
Store
6. Message
7. Message

73
Sender
Receiver
Store
8. Message'

74
Sender
Receiver
Store
8. Message'
9. Message'

Tying it all together

• Distributed Workers Example
– Front Ends receive requests
– Master receives work from Front Ends
– Workers pull work from Master
• Available as a Typesafe Activator template
– Zero configuration setup
– Code and tutorial
76

77
Worker
Master
Worker
Worker
Worker
Frontend
Frontend

• Further Goals
– Elastic addition/removal of front end nodes
– Elastic addition/removal of workers
– Thousands of workers
– Jobs should not be lost
78

79
Master
Worker
Frontend
Work
Executor

80
Master
Worker
Frontend
Work
Executor
RegisterWorker

81
Master
Worker
Frontend
1. Work
Work
Executor

82
Master
Worker
Frontend
1. Work
2. Work
Work
Executor

83
Master
Worker
Frontend
1. Work
2. Work
3. Ack
Work
Executor

84
Master
Worker
Frontend
1. Work
2. Work
3. Ack
4. OK
Work
Executor

85
Master
Worker
Frontend
1. Work
2. Work
3. Ack
4. OK
5. WorkIsReady
Work
Executor

86
Master
Worker
Frontend
1. Work
2. Work
3. Ack
4. OK
5. WorkIsReady
6. WorkerRequestWork
Work
Executor

87
Master
Worker
Frontend
1. Work
2. Work
3. Ack
4. OK
5. WorkIsReady
7. Work
Work
Executor

88
Master
Worker
Frontend
1. Work
2. Work
3. Ack
4. OK
5. WorkIsReady
7. Work
Work
Executor
8. Work

89
Master
Worker
Frontend
Work
Executor

90
Master
Worker
Frontend
Work
Executor
9. WorkComplete

91
Master
Worker
Frontend
Work
Executor
9. WorkComplete
10. WorkIsDone

92
Master
Worker
Frontend
Work
Executor
9. WorkComplete
10. WorkIsDone
11. Ack

Cluster Technologies/Patterns
• Distributed Pub/Sub Mediator
– Publish and Subscribe to message flows
• Cluster Singleton
– HA singleton actor instance within the cluster
• Cluster Client
– Let other systems connect to the cluster
93

DistributedPubSubMediator
94
Frontend Master
Mediator Mediator

95
Frontend Master
Mediator Mediator
Put

96
Frontend Master
Mediator Mediator
Send

97
Frontend Master
Mediator Mediator
Send

ClusterSingleton
98
ClusterSingleton
Manager
ClusterSingleton
Manager
Master
Master
(Standby)

ClusterSingleton
99
ClusterSingleton
Manager
ClusterSingleton
Manager
Master
Master
(Standby)

ClusterSingleton
100
ClusterSingleton
Manager
ClusterSingleton
Manager
Master Master

ClusterClient & ClusterSingleton
101
Master
Mediator
Mediator
Receptionist
Master
(Standby)
Receptionist
Cluster
Client
Cluster
Client
Worker
Worker

102
Master
Mediator
Mediator
Receptionist
Master
(Standby)
Receptionist
Cluster
Client
Cluster
Client
Worker
Worker
Cluster
Client
Worker
Worker
Frontend
Frontend

Resources

Resources: Blog Posts
• Push with rate limiting
– Kaspar Fischer
http://letitcrash.com/post/28901663062/throttling-messages-in-akka-2
• Pull
– Derek Wyatt
http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-
akka-2
– Michael Pollmeier
http://www.michaelpollmeier.com/akka-work-pulling-pattern-to-throttle-work/
104

Resources: Typesafe Activator
• http://typesafe.com/platform/getstarted
105

Resources: Coursera Course
• Principles of Reactive Programming by
Martin Odersky, Erik Meijer and Roland Kuhn
– Starts 4th of November 2013
– 7 weeks
– Workload: 5-7 hours a week
– Free as in free beer
• https://www.coursera.org/course/reactive
106

Björn Antonsson
bjorn.antonsson@typesafe.com
@bantonsson

Real World Akka Actor Recipes JavaOne 2013

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Real World Akka Actor Recipes JavaOne 2013