Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
real-time delivery architecture

@raffi
qcon shanghai 2013
@raffi
qcon shanghai 2013
http://twitpic.com/135xa There’s a plane in the
Hudson. I’m on a ferry
going to pick up the
people. Crazy.

!
15 Jan 09
Ja...
what are the goals?
⇢ evolve from being solely a web stack

@raffi
qcon shanghai 2013
@raffi
qcon shanghai 2013
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Gizmoduck

Flock(s)

@raffi
qcon shangha...
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Gizmoduck

Flock(s)

@raffi
qcon shangha...
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Gizmoduck

Flock(s)

@raffi
qcon shangha...
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Gizmoduck

Flock(s)

@raffi
qcon shangha...
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Gizmoduck

Flock(s)

@raffi
qcon shangha...
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Gizmoduck

Flock(s)

@raffi
qcon shangha...
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Darkwing

Flock(s)

@raffi
qcon shanghai...
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Darkwing

Flock(s)

@raffi
qcon shanghai...
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Darkwing

Flock(s)

@raffi
qcon shanghai...
ROUTING

PRESENTATION

LOGIC

STORAGE &
RETRIEVAL

T-Bird

T-Flock +
Haplo
Monorail
Darkwing

Flock(s)

@raffi
qcon shanghai...
what are the goals?
⇢ evolve from being solely a web stack
⇢ isolate responsibilities and concerns
⇢ site speed and reliab...
@raffi
qcon shanghai 2013
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
@raffi
qcon shanghai 2013
@raffi
qcon shanghai 2013
@raffi
qcon shanghai 2013
@raffi
qcon shanghai 2013
@raffi
qcon shanghai 2013
Write API

Ingester

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis

Timeline Cache

Search Cache

Redis

Fanout

@...
Write API

Ingester

@raffi
qcon shanghai 2013
Write API

@raffi
qcon shanghai 2013
Write API

Ingester

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cache

Searc...
Write API

Ingester

Hadoop

Batch Compute

Push Compute

Search Cache

Fanout

@raffi
qcon shanghai 2013
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Blender

HTTP Push

Mobile
Push

Hadoop

Batch Compute

Redis
Redis
Redis

Timeline Cache

Search Cache

Redis
...
Write API

⇢ pipelined 4k
“destinations”
at a time
Blender

⇢ replicated

HTTP Push

Mobile
Push

Hadoop

Batch Compute

⇢...
Write API

Ingester

Blender

⇢ RPUSHX to
only add to
cached
timelines

Tweet IDPush User ID
HTTP
8 bytes
8 bytes
Mobile
P...
Write API

Ingester

Blender

⇢ RPUSHX to
only add to
cached
timelines

Tweet IDPush User ID
HTTP

Bits

Tweet ID

Bits Ha...
Write API

Ingester

Blender

⇢ RPUSHX to
only add to
cached
timelines

Tweet IDPush User ID
HTTP

Bits

Tweet ID

Bits Ha...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Timeline
Service

HTTP Push

Hadoop

⇢ queries one
replica of all
indexes
Mobile
Push

Batch...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
http push / hosebird
⇢ maintains persistent connections with
end clients
⇢ processes tweet & social graph events
⇢ event-b...
Hosebird
Write API

Hosebird

Firehose
User Streams

Hosebird

Track / Follow

Text

@raffi
qcon shanghai 2013
Hosebird
Write API

Hosebird

Firehose
User Streams

Hosebird

Track / Follow

@raffi
qcon shanghai 2013
Hosebird
Write API

Hosebird

Firehose
User Streams

Hosebird

Track / Follow

event propagation
⇢ write API sends all eve...
Hosebird
Write API

Hosebird

Firehose
User Streams

Hosebird

Track / Follow

@raffi
qcon shanghai 2013
Hosebird
Write API

Hosebird

Firehose
User Streams

Hosebird

Track / Follow

event cascading
⇢ bandwidth management
⇢ si...
Hosebird
Write API

Hosebird

Firehose
User Streams

Hosebird

Track / Follow

@raffi
qcon shanghai 2013
Hosebird
Write API

Hosebird

Firehose
User Streams

Hosebird

Track / Follow

firehose
⇢ edge machine simply outputs the p...
Hosebird
Write API

Hosebird

Firehose
Track / Follow

Hosebird

User Streams

@raffi
qcon shanghai 2013
Hosebird
Write API

Hosebird

Firehose
Track / Follow

Hosebird

User Streams

track / follow
⇢ simple query based on twee...
Hosebird
Write API

Hosebird

Firehose
Track / Follow

Hosebird

User Streams

@raffi
qcon shanghai 2013
Hosebird
Write API

Hosebird

Firehose
Track / Follow

Hosebird

User Streams

user streams
⇢ replicate home timeline expe...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop
Social
Graph
Service

Batch Compute

HTTP Push

Push Compute

Redis
Redi...
Write API

Ingester

Blender

Mobile
Push

Hadoop
Social
Graph
Service

Batch Compute

HTTP Push

Push Compute

Redis
Redi...
Write API

Ingester

Blender

Mobile
Push

Hadoop
Social
Graph
Service

Batch Compute

HTTP Push

Push Compute

Redis
Redi...
Write API

Ingester

Blender

Mobile
Push

Hadoop
Social
Graph
Service

Batch Compute

HTTP Push

Push Compute

Redis
Redi...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Pull
Targeted twitter.com
home_timeline API

Queried Search API

Push
User / Site Streams
Mobile Push (SMS, etc.)
Track / ...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Synchronous Path
Write API

Ingester

Blender

Timeline
Service

Mobile
Push

Asynchronous Path
Hadoop

Batch Compute

HTT...
Synchronous Path
Write API

Ingester

Blender

Timeline
Service

Mobile
Push

Asynchronous Path
Hadoop

Batch Compute

HTT...
Synchronous Path
Write API

Ingester

Blender

Timeline
Service

Mobile
Push

Asynchronous Path
Hadoop

Batch Compute

HTT...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
Blender

Write Path

HTTP Push

Mobile
Push

Hadoop

Batch Compute

Redis
Redis
Redis

Timeline Cache

Redis
Earlybird
Ear...
Blender

Write Path

HTTP Push

Mobile
Push

Hadoop

Batch Compute

Redis
Redis
Redis

Timeline Cache

Redis
Earlybird
Ear...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
things we’re trying...

@raffi
qcon shanghai 2013
@raffi
qcon shanghai 2013
Write API

Redis
Earlybird
Earlybird

Fanout

Search Index

Ingester

Redis
Redis
Redis

@raffi
qcon shanghai 2013
Write API

Redis
Redis
Redis

Timeline Cache

Redis
Earlybird
Earlybird

Fanout

Search Index

Ingester

@raffi
qcon shangha...
Write API

Redis
Redis
Redis

Timeline Cache

Redis
Earlybird
Earlybird

Fanout

Search Index

Ingester

@raffi
qcon shangha...
Write API

Redis
Redis
Redis

Timeline Cache

Redis
Earlybird
Earlybird

Fanout

Search Index

Ingester

@raffi
qcon shangha...
Write API

Redis
Redis
Redis

Timeline Cache

Redis
Earlybird
Earlybird

Fanout

Search Index

Ingester

@raffi
qcon shangha...
Write API

Redis
Redis
Redis

Timeline Cache

Redis
Earlybird
Earlybird

Fanout

Search Index

Ingester

@raffi
qcon shangha...
Write API

Redis
Redis
Redis

Timeline Cache

Redis
Earlybird
Earlybird

Fanout

Search Index

Ingester

@raffi
qcon shangha...
Write API

search index
⇢[‘hello’,‘world’]

Redis
Redis
Redis

Timeline Cache

Redis
Earlybird
Earlybird

Fanout

Search I...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi

@raffi
qcon shan...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi

@raffi
qcon shan...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi

@raffi
qcon shan...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi

@raffi
qcon shan...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi

@raffi
qcon shan...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi

@raffi
qcon shan...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi

@raffi
qcon shan...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi

@raffi
qcon shan...
Write API

fan-in
⇢ O(n) read
Redis
Earlybird
Earlybird

⇢ O(n) write

Fanout

Redis
Redis
Redis

Timeline Cache

Ingester...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi

@raffi
qcon shan...
User Intent

Query Expansion

“Hello, world”

“Hello” AND “world”

@raffi’s home timeline

home_timeline:raffi
OR
user_timelin...
streaming compute
⇢ continuous computation
⇢ driven by the events that come into
twitter
⇢ generalizing the push mechanism...
Write API

Ingester

Blender

Mobile
Push

Hadoop

Batch Compute

HTTP Push

Push Compute

Redis
Redis
Redis

Timeline Cac...
timeline query statistics
⇢ >150m active users worldwide
⇢ 300k qps poll-based timelines 

@ 1ms p50 / 4ms p99
⇢ 30k qps s...
tweet input
⇢ ~340m tweets per day
⇢ ~4K/sec daily average
⇢ ~6K/sec daily peak
⇢ >10K/sec during large events
@raffi
qcon s...
@raffi
qcon shanghai 2013
followed by

following

@raffi
qcon shanghai 2013
@raffi
qcon shanghai 2013
timeline delivery statistics
⇢ 26b deliveries / day (~18m / min)
⇢ 3.5 seconds @ p50 to deliver to 1m
⇢ ~300k deliveries /...
thanks!
Upcoming SlideShare
Loading in …5
×

Q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]

  • Be the first to comment

  • Be the first to like this

Q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]

  1. 1. real-time delivery architecture @raffi qcon shanghai 2013
  2. 2. @raffi qcon shanghai 2013
  3. 3. http://twitpic.com/135xa There’s a plane in the Hudson. I’m on a ferry going to pick up the people. Crazy. ! 15 Jan 09 Janis Krums @jkrums @raffi qcon shanghai 2013
  4. 4. what are the goals? ⇢ evolve from being solely a web stack @raffi qcon shanghai 2013
  5. 5. @raffi qcon shanghai 2013
  6. 6. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  7. 7. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  8. 8. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  9. 9. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  10. 10. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  11. 11. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  12. 12. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
  13. 13. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
  14. 14. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
  15. 15. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
  16. 16. what are the goals? ⇢ evolve from being solely a web stack ⇢ isolate responsibilities and concerns ⇢ site speed and reliability ⇢ developer innovation speed @raffi qcon shanghai 2013
  17. 17. @raffi qcon shanghai 2013
  18. 18. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  19. 19. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  20. 20. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  21. 21. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  22. 22. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  23. 23. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  24. 24. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  25. 25. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  26. 26. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  27. 27. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  28. 28. @raffi qcon shanghai 2013
  29. 29. @raffi qcon shanghai 2013
  30. 30. @raffi qcon shanghai 2013
  31. 31. @raffi qcon shanghai 2013
  32. 32. @raffi qcon shanghai 2013
  33. 33. Write API Ingester Hadoop Batch Compute HTTP Push Push Compute Redis Timeline Cache Search Cache Redis Fanout @raffi qcon shanghai 2013
  34. 34. Write API Ingester @raffi qcon shanghai 2013
  35. 35. Write API @raffi qcon shanghai 2013
  36. 36. Write API Ingester Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout @raffi qcon shanghai 2013
  37. 37. Write API Ingester Hadoop Batch Compute Push Compute Search Cache Fanout @raffi qcon shanghai 2013
  38. 38. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  39. 39. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  40. 40. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  41. 41. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  42. 42. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  43. 43. Write API Blender HTTP Push Mobile Push Hadoop Batch Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Push Compute Ingester Social Graph Service Timeline Service @raffi qcon shanghai 2013
  44. 44. Write API ⇢ pipelined 4k “destinations” at a time Blender ⇢ replicated HTTP Push Mobile Push Hadoop Batch Compute ⇢ keyed off “recipient” Redis Redis Redis Timeline Cache Redis Redis Earlybird Search Cache insert Fanout Push Compute Ingester Social Graph Service Timeline Service @raffi qcon shanghai 2013
  45. 45. Write API Ingester Blender ⇢ RPUSHX to only add to cached timelines Tweet IDPush User ID HTTP 8 bytes 8 bytes Mobile Push Bits Hadoop 4 bytes Batch Compute ⇢ native list structure Redis Redis Redis Push Compute Redis Redis Earlybird Search Cache using redis Timeline Cache Fanout Timeline Service @raffi qcon shanghai 2013
  46. 46. Write API Ingester Blender ⇢ RPUSHX to only add to cached timelines Tweet IDPush User ID HTTP Bits Tweet ID Bits Hadoop User ID Bits Tweet ID User ID Bits Tweet ID Timeline Service Mobile Tweet ID User ID Push User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Batch Compute ⇢ native list structure Redis Redis Redis Push Compute Redis Redis Earlybird Search Cache using redis Timeline Cache Fanout Bits @raffi qcon shanghai 2013
  47. 47. Write API Ingester Blender ⇢ RPUSHX to only add to cached timelines Tweet IDPush User ID HTTP Bits Tweet ID Bits Hadoop ID Tweet User ID Bits Tweet ID User ID Bits Tweet ID Timeline Service Mobile Tweet ID User ID Push User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Batch Compute ⇢ native list structure Redis Redis Redis Push Compute Redis Redis Earlybird Search Cache using redis Timeline Cache Fanout Bits Tweet ID Tweet ID @raffi qcon shanghai 2013
  48. 48. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  49. 49. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  50. 50. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  51. 51. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  52. 52. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  53. 53. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  54. 54. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  55. 55. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  56. 56. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  57. 57. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  58. 58. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  59. 59. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  60. 60. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  61. 61. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  62. 62. Write API Ingester Blender Timeline Service HTTP Push Hadoop ⇢ queries one replica of all indexes Mobile Push Batch Compute blender Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout ⇢ merges & ranks results @raffi qcon shanghai 2013
  63. 63. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  64. 64. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  65. 65. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  66. 66. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  67. 67. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  68. 68. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  69. 69. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  70. 70. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  71. 71. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  72. 72. http push / hosebird ⇢ maintains persistent connections with end clients ⇢ processes tweet & social graph events ⇢ event-based “router” @raffi qcon shanghai 2013
  73. 73. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow Text @raffi qcon shanghai 2013
  74. 74. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow @raffi qcon shanghai 2013
  75. 75. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow event propagation ⇢ write API sends all events into hosebird; sees content creation events, social graph changes, etc. ⇢ different queues for public tweets, protected tweets, social events, etc. @raffi qcon shanghai 2013
  76. 76. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow @raffi qcon shanghai 2013
  77. 77. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow event cascading ⇢ bandwidth management ⇢ simultaneous connection management (~1m long lived & open connections to this cluster) @raffi qcon shanghai 2013
  78. 78. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow @raffi qcon shanghai 2013
  79. 79. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow firehose ⇢ edge machine simply outputs the public tweet queue ⇢ only allow a limited number of firehoses per hosebird box for bandwidth management @raffi qcon shanghai 2013
  80. 80. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams @raffi qcon shanghai 2013
  81. 81. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams track / follow ⇢ simple query based on tweet content ⇢ keeps list of terms / users of interest ⇢ parses public tweets at the edge, and if term matches a token, or user is of interest, then route @raffi qcon shanghai 2013
  82. 82. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams @raffi qcon shanghai 2013
  83. 83. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams user streams ⇢ replicate home timeline experience ⇢ upon login, obtain “following” list ⇢ keep cached following list coherent by seeing social graph updates ⇢ route tweet if from a followed user @raffi qcon shanghai 2013
  84. 84. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  85. 85. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  86. 86. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  87. 87. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  88. 88. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  89. 89. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  90. 90. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  91. 91. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  92. 92. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  93. 93. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  94. 94. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  95. 95. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  96. 96. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  97. 97. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  98. 98. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  99. 99. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  100. 100. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  101. 101. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  102. 102. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  103. 103. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  104. 104. Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Query Path @raffi qcon shanghai 2013
  105. 105. Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Query Path @raffi qcon shanghai 2013
  106. 106. Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Query Path @raffi qcon shanghai 2013
  107. 107. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  108. 108. Blender Write Path HTTP Push Mobile Push Hadoop Batch Compute Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Push Compute Ingester Search Index Read Path Write API Timeline Service @raffi qcon shanghai 2013
  109. 109. Blender Write Path HTTP Push Mobile Push Hadoop Batch Compute Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Push Compute Ingester Search Index Read Path Write API Timeline Service @raffi qcon shanghai 2013
  110. 110. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  111. 111. things we’re trying... @raffi qcon shanghai 2013
  112. 112. @raffi qcon shanghai 2013
  113. 113. Write API Redis Earlybird Earlybird Fanout Search Index Ingester Redis Redis Redis @raffi qcon shanghai 2013
  114. 114. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  115. 115. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  116. 116. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  117. 117. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  118. 118. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  119. 119. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  120. 120. Write API search index ⇢[‘hello’,‘world’] Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester fanout index ⇢[@danadanger, ...] @raffi qcon shanghai 2013
  121. 121. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  122. 122. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  123. 123. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  124. 124. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  125. 125. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  126. 126. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  127. 127. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  128. 128. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  129. 129. Write API fan-in ⇢ O(n) read Redis Earlybird Earlybird ⇢ O(n) write Fanout Redis Redis Redis Timeline Cache Ingester Search Index ⇢ O(1) write fan-out ⇢ O(1) read @raffi qcon shanghai 2013
  130. 130. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  131. 131. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi OR user_timeline:taylorswift13 @raffi qcon shanghai 2013
  132. 132. streaming compute ⇢ continuous computation ⇢ driven by the events that come into twitter ⇢ generalizing the push mechanism @raffi qcon shanghai 2013
  133. 133. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  134. 134. timeline query statistics ⇢ >150m active users worldwide ⇢ 300k qps poll-based timelines 
 @ 1ms p50 / 4ms p99 ⇢ 30k qps search-based timelines @raffi qcon shanghai 2013
  135. 135. tweet input ⇢ ~340m tweets per day ⇢ ~4K/sec daily average ⇢ ~6K/sec daily peak ⇢ >10K/sec during large events @raffi qcon shanghai 2013
  136. 136. @raffi qcon shanghai 2013
  137. 137. followed by following @raffi qcon shanghai 2013
  138. 138. @raffi qcon shanghai 2013
  139. 139. timeline delivery statistics ⇢ 26b deliveries / day (~18m / min) ⇢ 3.5 seconds @ p50 to deliver to 1m ⇢ ~300k deliveries / sec @raffi qcon shanghai 2013
  140. 140. thanks!

×