Presentation of Finagle, Twitter's RPC library, ScalaIO 2014.

1. Finagle: Twitter’s RPC Library
24 Oct. 2014
Steve Gury
@stevegury

2. > whoami
stevegury
2005 2008 2011 2014 time
Distributed Systems
Massively Multiplayer Online Games
Twitter
Scala

3. Agenda
1. Motivation
2.Finagle TL;DR
3.Three Key Abstractions
4.Finagle Big Picture

4. Motivation & Context
2006 2010 2014
>2M Tweets / day
(January 2009)
>65M Tweets / day
(July 2010)
>500M Tweets / day
(August 2013)

5. Agenda
1. Motivation
2.Finagle TL;DR
3.Three Key Abstractions
4.Finagle Big Picture

6. Finagle TL;DR
Compose RPC like you
compose functions

7. Composing Functions
getUserId(name: String): Int getTweets(userId: Int): Tweets
getTweets(name: String): Tweets
= getUserId ◦ getTweets (name)

8. Agenda
1. Motivation
2.Finagle TL;DR
3.Three Key Abstractions
4.Finagle Big Picture

9. Futures
Futures are containers for value
Pending Successful Failed
Present

10. Futures
a
val a: Future[Int]

11. Futures
a
b
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }

12. Futures
a
b
c
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }

13. Futures
a
b
c
d
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)

14. Futures
a
b
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

15. Futures
16 b
a
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

16. Futures
16 b
a
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

17. Futures
16
a
528
b
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

18. Futures
16
a
528
b
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

19. Futures
16
a
528
b
4
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

20. Futures
16
a
528
b
4
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

21. Futures
16
a
528
b
4
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

22. Futures
16
a
528
b
4
(528,
4)
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

23. Futures
16
a
528
b
4
(528,
4)
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

24. Futures
16
a
528
b 532
4
(528,
4)
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

25. Futures
0 b
a
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

26. Futures
0
a
512
b
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

27. Futures
0
a
512
b
Ex
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

28. Futures
0
a
512
b Ex
Ex
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

29. Futures
0
a
512
b Ex Ex
Ex
c
d e
val a: Future[Int]
val b: Future[Int] = a map { x => x + 512 }
val c = a map { x => 64 / x }
val d = Future.join(b,c)
val e = d map { case (x, y) => x + y }

30. Futures
Lots of combinators:
!
map -> add an edge+node in the graph
flatMap -> combine two graphs together
handle -> deal with exceptions
join -> merge multiple nodes of the graphs
select -> select the first node with a value
!
…more

31. Programming with Futures == Building a Graph

32. Service
Remember the key idea: treat RPC like functions

33. Service
Service is a Function from Request to Response
!
trait Service[Request, Response] {!
def apply(req: Request): Response!
}!
!
Equivalent to Scala’s Function1: Request=>Response

34. Service
But it has to be asynchronous, and treat exception as
value
!
trait Service[Request, Response] {!
def apply(req: Request): Future[Response]!
}!
!

35. Service
It also has to be closable for proper management of
resources.
!
trait Service[Request, Response] {!
def apply(req: Request): Future[Response]!
def close(): Future[Unit]!
}!

36. ServiceFactory
Factory of Service
(Function that returns a Function)

37. ServiceFactory
Function that asynchronously returns a Service
!
trait ServiceFactory[Req, Rep] {!
def apply(): Future[Service[Req, Rep]]!
def close(): Future[Unit]!
}

38. Agenda
1. Motivation
2.Finagle TL;DR
3.Three Key Abstractions
4.Finagle Big Picture

39. Transport
trait Transport[In, Out] {
def write(req: In): Future[Unit]
def read(): Future[Out]
}
Transport

40. Dispatcher
class Dispatcher[A, B] extends Service[…] {
def apply(req: A): Future[B]
}
Dispatcher Transport

41. Connection Pool
class ConnPool[A,B] extends ServiceFactory[…]{
def apply(): Future[Service[A, B]]
}
Connection Dispatcher Transport
Pool

42. Load Balancer
class LoadBalancer[A,B] extends ServiceFactory[…]{
def apply(): Future[Service[A, B]]
}
Connection Dispatcher Transport
Pool
Load
Balancer

43. Finagle in Action
Connection Dispatcher Transport
Pool
Load
Balancer
val req: Request
loadbalancer() // LB select an host
// Pool select a connection

44. Finagle in Action
val req: Request
loadbalancer() flatMap {
service => // LBService(PoolService(disp))
}
Connection Dispatcher Transport
Pool
Load
Balancer

45. Finagle in Action
val req: Request
loadbalancer() flatMap {
service => // LBService(PoolService(disp))
service(req) // disp write req on Transp.
}
Connection Dispatcher Transport
Pool
Load
Balancer

46. Finagle in Action
val req: Request
loadbalancer() flatMap {
service => // LBService(PoolService(disp))
service(req) ensure {
service.close() // close LBService
// close PoolService
}
}
Connection Dispatcher Transport
Pool
Load
Balancer

47. FactoryToService
class FactoryToService[A, B](factory: ServiceFactory[A, B])
extends Service[A, B]
{
def apply(request: A): Future[B] =
factory() flatMap { service =>
service(request) ensure {
service.close()
}
}
}
Connection Dispatcher Transport
Pool
Load
Balancer
FactoryTo
Service

48. Connection Dispatcher Transport
Pool
Load
Balancer
Finagle is “just” a composition of independent
functions on top of a Transport
FactoryTo
Service

49. The “Simplified” Full Picture
Stats
Timeout
Draining
Load Balancer
Monitor
Stats
Failure accrual
Timeout
Conn. Pool
Expiration Dispatcher
ServiceFactory
Service

50. The Stack is configured for you
You rarely have to configured the stack, we have a series
of predefined stack for you.
!
scala> val client: ServiceFactory[Req, Res] = Http.client!
.newClient(“localhost:8080")!
!

51. The Stack is configured for you
You rarely have to configured the stack, we have a series
of predefined stack for you.
!
scala> val client: Service[Req, Res] = Http.client!
.newClient(“localhost:8080")!
.toService!
scala> client(req)!

52. The Stack is configured for you
You rarely have to configured the stack, we have a series
of predefined stack for you.
!
scala> val client: Service[Req, Res] = Http.client!
.configured(DefaultPool.Param(5, 10, 0, 1.minute, 100))!
.newClient(“localhost:8080")!
.toService!
scala> client(req)!
!
You can configure any layer of the Stack

53. Finagle is doing more…
• Load balancing
• Connection pooling and request buffering
• Dynamic membership (Zookeeper)
• Failure detection and mitigation (fail-fast & failure
accrual)
• Statistics for visibility
• Distributed tracing (Zipkin)
• Cancellation propagation
• Automatic retrying
• Graceful shutdown and request draining
• GC avoidance - traffic shaping
• Backup requests

54. Load balancing

55. Load balancing
1
0
0
10

56. Load balancing
1
10
0

57. Load balancing
24
25
22
0

58. A Better Load Balancer
Key idea: Use history of recorded latencies to estimate
the “cost” of sending a request to a server

59. Experimental Latencies

60. A better Load Balancer
First idea: calculate an average of the latencies

61. Experimental Latencies

62. A better Load Balancer
Better idea: calculate EWMA average of the latencies
EWMA(n+1) = EWMA(n) * ⍺ + (1 - ⍺) * Xn+1

63. Experimental Latencies

64. A better Load Balancer
The devil is in the details
!
Recency of data: EWMA unevenly spaced t-Series
EWMA(n+1) = EWMA(n) * e-td/Tau + (1 - e-td/Tau) * Xn+1
No history: Initial cost=0 + Probation

65. Cost & Experimental Latencies

66. A Better Load Balancer
The devil is in the details
!
Speed of convergence: Cost “rides the peaks”
Outliers: Decaying cost

67. Cost & Experimental Latencies

68. A Better Load Balancer
The devil is in the details
!
CPU intensive: “The Power of 2 Choices”

69. A Better Load Balancer
“The Power of 2 Choices in Randomized Load Balancing” P.h.D
Thesis of M. Mitzenmacher

70. Load balancing
0 20
0
10
0
cost: 20ms
cost: 10ms
cost: 10ms

71. cost: 18ms
Load balancing
0 20
0
10
0
20ms
cost: 10ms
cost: 10ms

72. The “Simplified” Full Picture
Stats
Timeout
Draining
Load Balancer
Monitor
Stats
Failure accrual
Timeout
Conn. Pool
Expiration Dispatcher
ServiceFactory
Service

73. Finagle’s Architecture
Experience from writing “large scale” software
!
Independent modules
Simple contracts
Powerful combinators

74. Conclusion
• Finagle is a library that lets you treat RPC as functions
• Finagle is itself composed of independent functions
• Very widely used at Twitter, Foursquare, Tumblr,
Pinterest, SoundCloud…
• Protocol agnostic (thrift, http, redis, mysql…)
Github: github.com/twitter/finagle
“Your server as a Function” http://monkey.org/~marius/
funsrv.pdf

75. @finagle
@stevegury