Developing SEDA Based Applications using Akka.

1. Akka
Developing SEDA Based
Applications

2. Me
Ben Darfler
@bdarfler
http://bdarfler.com
Senior Software Engineer at Localytics

3. Localytics
Real time mobile analytics platform
40M+ events per day and growing rapidly
3x growth over the past 3 months
Heavy users of Scala/Akka/NoSql
We are hiring (seriously, come talk to me)

4. Localytics
How to keep up with our growth?

5. Actor Model
Lock free approach to concurrency
No shared state between actors
Asynchronous message passing
Mailboxes to buffer incoming messages

6. Akka
Configurable
● Dispatchers
● Mailboxes
Fault Tolerant
● Supervisors
Great community
● @jboner
● @viktorklang

7. Akka
Performant
http://blog.jayway.com/2010/08/10/yet-another-akka-benchmark/

8. SEDA
Staged Event Driven Architecture
"Decomposes a complex, event-driven
application into a set of stages connected
by queues." 1
"The most fundamental aspect of the SEDA
architecture is the programming model that
supports stage-level backpressure and load
management." 1
1. http://www.eecs.harvard.edu/~mdw/proj/seda/

9. Backpressure
Whats the big deal?

10. Backpressure
Manditory to prevent OutOfMemoryError
● Messages backup in memory faster than they
can be processed
Cassandra was seriously bitten by this
● Less crappy failure mode when swamped with
inserts than "run out of memory and gc-storm
to death" (CASSANDRA-401)
● Add backpressure to StorageProxy
(CASSANDRA-685)

11. Backpressure
Mailboxes
case class UnboundedMailbox(val blocking: Boolean = false) extends MailboxType
case class BoundedMailbox(
val blocking: Boolean = false,
val capacity: Int = {
if (Dispatchers.MAILBOX_CAPACITY < 0) Int.MaxValue
else Dispatchers.MAILBOX_CAPACITY
},
val pushTimeOut: Duration = Dispatchers.MAILBOX_PUSH_TIME_OUT
) extends MailboxType
Backpressure Mailbox
BoundedMailbox(false, QUEUE_SIZE, Duration(-1, "millis"))

12. Stages
How do we decompose the problem?

13. Stages
One actor class per stage
Shared dispatcher
Individually tunable
● I/O Bound
● CPU Bound
Easier to reason about
Code reuse

14. Dispatchers
ThreadBasedDispatcher
● Binds one actor to its own thread
ExecutorBasedEventDrivenDispatcher
● Must be shared between actors
ExecutorBasedEventDrivenWorkStealingDispatcher
● Must be shared between actors of the same type

15. Queues
SEDA has a queue per stage model
Akka actors have their own mailbox
How do we evenly distribute work?

16. Work Stealing
ExecutorBasedEventDrivenWorkStealingDispatcher
"Actors of the same type can be set up to share this
dispatcher and during execution time the different
actors will steal messages from other actors if they
have less messages to process" 1
1. http://doc.akka.io/dispatchers-scala

17. Work Stealing
Really a work "donating" dispatcher
"I have implemented a work stealing dispatcher for
Akka actors. Although its called "work stealing" the
implementation actually behaves more as "work
donating" because the victim actor takes the initiative.
I.e. it actually donates work to its thief, rather
than having the thief steal work from the victim." 1
1. http://janvanbesien.blogspot.com/2010/03/load-balancing-actors-with-work.html

18. Work Stealing
Doesn't that conflict with blocking mailboxes?

19. Work Stealing
Sending actor will block on the receiving actors
mailbox before it can "donate"
Might be fixed in Akka 1.1
● I owe @viktorklang a test of his latest changes

20. Load Balancing
Can we distribute work on the sender side?

21. Load Balancing
Routing.loadBalancerActor()
● Creates a new actor that forwards
messages in a load balancing fashion
InfiniteIterator
● CyclicIterator
● SmallestMailboxFirstIterator

22. Load Balancing
Doesn't the load balancer need a blocking mailbox?

23. Load Balancing
Can't easily change the load balancer's mailbox
Use SmallestMailboxFirstIterator directly
new SmallestMailboxFirstIterator(List(actor, actor, actor))

24. Fault Tolerance
Supervisors
● Restarts actors
● Stops after x times within y milliseconds
Restart Strategies
● OneForOne
● AllForOne

25. Fault Tolerance
Great for transient issues
● Network failures
Not great for permanent issues
● OutOfMemoryError

26. Final Product
// Actor creation
val supervisor = Supervisor(SupervisorConfig(
OneForOneStrategy(List(classOf[Exception]), RETRIES, WITH_IN_TIME),
Supervise(myActors))
def
myActors
: List[Supervise] = {
val mailbox = BoundedMailbox(false, QUEUE_SIZE, Duration(-1, "millis"))
val dispatcher =
Dispatchers.newExecutorBasedEventDrivenDispatcher(
"my-dispatcher", 1, mailbox).setCorePoolSize(POOL_SIZE).build
(1 to POOL_SIZE toList).foldRight(List[Supervise]()) {
(i, list) =>
Supervise(actorOf(new MyActor("my-actor-" + i, dispatcher)), Permanent) :: list
}
}
// Sending a message
val actors = new SmallestMailboxFirstIterator(actorsFor(classOf[MyActor]).toList)
def actor = actors.next
actor ! Message()

27. Thanks
@bdarfler
http://bdarfler.com