Our talk covers the migration of the Twitter architecture from primarily Ruby on Rails (RoR) to a JVM-based SOA system with emphasis on high performance, scalability, and resilience to failure. General lessons include the advantages of asynchronous, real-time architectures over synchronous, process / thread-oriented systems, as well as caching and data store patterns.

1. real-time systems @twitter
@raffi & @a_a
velocity 2012

2. ROUTING PRESENTATION LOGIC STORAGE &
RETRIEVAL
T-Bird
T-Flock +
Haplo
Monorail
Darkwing
Flock(s)

4. what are the big problems?
⇢ monolithic application
⇢ lack of self-service infrastructure
⇢ painful to add new services & features

5. what did we want to achieve?
⇢ big infrastructure wins in speed,
efficiency, reliability
⇢ separation of concerns
⇢ team independence

7. stats.timeFuture("request_latency_ms") {
// dispatch to do work
}

8. ts(
AVG,
timelineservice,
audubon.role.timelineservice,
service/client/
woodstar.prod/
getStatusTimeline/request_latency_ms.p50
)

10. TFE
(HTTP Proxy)
Woodstar

11. TFE
(HTTP Proxy)
Monorail Woodstar

12. 100% Monorail TFE
(HTTP Proxy) 0% Woodstar
Monorail Woodstar

13. 0% Monorail TFE
(HTTP Proxy) 100% Woodstar
Monorail Woodstar

16. TFE
(HTTP Proxy)
Woodstar

17. TFE
(HTTP Proxy)
Woodstar
Tweetypie Gizmoduck
Timeline
(Tweet (User
Service
Service) Service)

18. network substrate
⇢ connection management
⇢ protocol codecs
⇢ transient error handling
⇢ service discovery
⇢ observability

19. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

20. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

21. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

22. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

23. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

24. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

25. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

26. TFE
(HTTP Proxy)
Woodstar
Tweetypie Gizmoduck
Timeline
(Tweet (User
Service
Service) Service)

27. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

30. TFE
(HTTP Proxy)
Monorail Woodstar

31. TFE
(HTTP Proxy)
Macaw Macaw Macaw
Woodstar
+Activity +Search +Logging

33. class EchoLoadTest(service: ParrotThriftService)
extends RecordProcessor {
val client =
new EchoService.ServiceToClient(
service,
new TBinaryProtocol.Factory())
def processLines(
job: ParrotJob,
lines: Seq[String]) {
lines.map(client.echo(_))
}
}

35. TFE
(HTTP Proxy)
Monorail

36. TFE
(HTTP Proxy)
Monorail Woodstar

37. ROUTING PRESENTATION LOGIC STORAGE &
RETRIEVAL
T-Bird
T-Flock +
Haplo
Monorail
Darkwing
Flock(s)

38. ROUTING PRESENTATION LOGIC STORAGE &
RETRIEVAL
Tweetypie
Monorail T-Bird
Gizmoduck
T-Flock +
Woodstar
Haplo
TFE TLS
Macaw
Darkwing
+Swift
Social
Graph
Service
Macaw
Flock(s)
+Disco
Story
Service

39. where are we?
⇢ team organization that mimics the
software stack
⇢ able to launch massive features in
parallel

40. mentions statuses/show users/show
3000 500 400
2250 375 300
1500 250 200
750 125 100
0 0 0
p50 p95 p999 p50 p95 p999 p50 p95 p999

41. some more statistics
⇢ 45% of traffic on the JVM stack
⇢ we’re a lot faster
⇢ we’re a lot more reliable
⇢ we fix bugs faster
⇢ 12 deploys, yesterday

42. #JoinTheFlock