Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Streaming a million likes/second Real-time interactions on Live Video Akhilesh Gupta Realtime Engineer, LinkedIn 2
InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentati...
Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practi...
tiny.cc/qconlive 3 Login with LinkedIn credentials
Which was the largest live stream in the world? (largest number of concurrent viewers) ? 4
INDIA VS NEW ZEALAND WORLD CUP Largest Live Stream >25.3M viewers © The Indian Express https://images.indianexpress.com/20...
6
D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 7
8
What is a Live Video? 9
What are real-time interactions on Live Video? 10
Mobile LINKEDIN LIVE Likes/Comments Concurrent Viewer Counts 11
LINKEDIN LIVE Desktop Likes/Comments Concurrent Viewer Counts 12
Distribution of Likes A S 13
Distribution of Likes A S Simple HTTP Request 14
Distribution of Likes A S Simple HTTP Request ? 15
Challenge 1:The delivery pipe 16
The delivery pipe Real-time Delivery A S Likes Backend (Publisher) 17
The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Simple HTTP Request 18
The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Persistent Connection 19
The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Persistent Connection 20
Persistent Connection Real-time Delivery 21
TRY IT! Login at linkedin.com Persistent Connection tiny.cc/realtime 22
23
EVENTSOURCE INTERFACE CLIENT REQUEST HTTP Long Poll with Server Sent Events GET /connect HTTP/1.1 Accept: text/event-strea...
EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-...
EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-...
EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-...
EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-...
WEB EventSource Client Libraries var evtSource = new EventSource("https://www.linkedin.com/realtime/connect"); evtSource.o...
ANDROID & IOS SUPPORT EventSource Client Libraries Android: https://github.com/tylerjroach/eventsource-android iOS Swift: ...
Next Challenge? Real-time Delivery 31
Next Challenge: Multiple Connections Real-time Delivery 32
Challenge 2: Connection Management 33
PLAY & AKKA Connection Management Akka is a toolkit for building highly concurrent, distributed, and resilient message-dri...
© Disney, All Rights Reserved, Pirates of the Caribbean 35
AN AKKA ACTOR Connection Management State Behavior 36
AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor 37
AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor 38
AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor Millions of Actors 39
AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor No. of threads proportional to no. of cores M...
AN AKKA ACTOR Connection Management Connection Publish Event ABC Events to be published Millions of Connections 41
INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned con...
INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned con...
INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned con...
Publish Events Using Akka Actors Akka Actor Real-time Delivery System Connection ID cid1 45
EVENT PUBLISH FROM PUBLISHER Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Ac...
EVENT PUBLISH FROM SUPERVISOR TO CHILD ACTORS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 A...
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System A...
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publishing on EventSource eventSource.send(<like object>); 49
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System A...
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System A...
EVENTSOURCE INTERFACE Received Events on the Clients HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream ...
Next Challenge? Real-time Delivery 53
Next Challenge: Multiple Live Videos Real-time Delivery 54
Challenge 3:Multiple Live Videos 55
START WATCHING LIVE VIDEO Multiple Live Videos cid3 cid5 Live Video 1 Live Video 2 Real-time Delivery System 56
HTTP SUBSCRIPTION REQUEST Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid3 cid5 Live...
IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid5 Live V...
IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid3 cid5 L...
Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 L...
PUBLISHER PUBLISH TO SUPERVISOR ACTOR Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 c...
SUBSCRIBER LOOKUP FROM IN MEMORY TABLE Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 ...
PUBLISH TO CLIENT CONNECTIONS Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3...
PUBLISHER PUBLISH TO SUPERVISOR ACTOR Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 c...
SUBSCRIBER LOOKUP FROM IN MEMORY TABLE Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 ...
PUBLISH TO CLIENT CONNECTIONS Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3...
Next Challenge? Real-time Delivery 67
Next Challenge: 10K Concurrent Viewers Real-time Delivery 68
Challenge 4:10K Concurrent Viewers 69
70
10K Concurrent Viewers Real-time Delivery 71
S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 72
10K Concurrent Viewers Real-time Delivery 73
10K Concurrent Viewers Frontend Frontend 74
10K Concurrent Viewers Frontend Frontend 75
DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 76
DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 77
DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 78
Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Real-time Dispatcher 79 Live Video 1
HTTP SUBSCRIPTION REQUEST Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Fronte...
SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Fronte...
SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Fronte...
SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Fronte...
Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend ...
PUBLISHER PUBLISH TO DISPATCHER Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Fronten...
SUBSCRIBER LOOKUP Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Fronte...
PUBLISH TO SUBSCRIBED FRONTEND NODES Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Fr...
Frontend Node to Client Publish Frontend 88
Frontend Node to Client Publish Frontend 89
What’s the next challenge? Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 L...
Challenge 5:100 Likes/second 91
100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video ...
S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 93
100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video ...
100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video ...
100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video ...
100 Likes/second Frontend Frontend Frontend Frontend Frontend Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Liv...
COUCHBASE/REDIS ETC. Key-Value Store for Subscriptions Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher ...
Publish to Dispatcher Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value...
Subscriber lookup from KV Store Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-V...
Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value S...
Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value S...
Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value S...
Challenge 6: 100 Likes/s, 10K Viewers Distribution of 1M Likes/s 104
Mobile LINKEDIN LIVE Likes 105
The Subscription Flow Frontend Dispatcher Key-Value Store 1 106
START WATCHING LIVE VIDEO Viewers subscribing to Live Video Likes cid3 cid5 Live Video 1 Live Video 2 Frontend 107
HTTP SUBSCRIPTION REQUEST Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /...
FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid5 Live Video 2 /subscr...
FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Li...
FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Li...
The Subscription Flow Frontend Dispatcher Key-Value Store 1 2 3 112
FRONTEND RECEIVES SUBSCRIPTION REQUEST Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Fro...
FRONTEND HTTP SUBSCRIPTION REQUEST TO DISPATCHER Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live V...
DISPATCHER SUBSCRIPTION ENTRY IN KV STORE Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 ...
DISPATCHER SUBSCRIPTION ENTRY IN KV STORE Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 ...
The Publish Flow Frontend Dispatcher Key-Value Store 1 117
Viewers liking Live Videos Likes Backend (Publisher) 118
Likes Published to Backend Likes Backend (Publisher) 119
The Publish Flow Frontend Dispatcher Key-Value Store 1 2 345 120
The Publish Flow Frontend Dispatcher Key-Value Store 2 345 121
BACKEND PUBLISH TO DISPATCHER NODES Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Disp...
SUBSCRIBER LOOKUP FROM KV STORE Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatch...
SUBSCRIBER LOOKUP FROM KV STORE Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatch...
PUBLISH TO FRONTEND NODES Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dis...
The Publish Flow Frontend Dispatcher Key-Value Store 5 126
DISPATCHER PUBLISH TO FRONTEND NODE SUPERVISOR ACTOR Frontend Publish to Mobile Clients Actor 1 Actor 2 Actor 3 Actor 4 Ac...
SUBSCRIBED CLIENT CONNECTION LOOKUP FROM IN-MEMORY TABLE Frontend Publish to Mobile Clients Actor 1 Frontend Actor 2 Actor...
PUBLISH TO MOBILE CLIENTS Frontend Publish to Mobile Clients Actor 1 Frontend Actor 2 Actor 3 Actor 4 Actor 5 Supervisor c...
Bonus Challenge: Another data center 130
131
D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 132
Frontend DispatcherDC-1 133
Frontend Dispatcher Frontend DispatcherDC-2 DC-1 134
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 135
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 136
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 137
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 138
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 CrossDataCenterPublish 139
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 140
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 141
Performance & Scale 142
How many persistent connections can a single machine hold? Frontend Server ? 143
> 100,000 144
ROYAL WEDDING Second Largest Stream >18M viewers © time.com 145
> 100,000 We can hold those 18M connections with just 180 machines 146
LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinscaling >100K 147
LinkedIn Realtime Performance Persistent Connections/machine 5K/s Dispatcher Publish QPS/machine 75ms E2E Publish Latency ...
Dispatcher Performance Frontend Frontend Frontend Frontend Frontend Dispatcher How many events per second can be published...
Dispatcher Performance Number of frontend nodes to publish to is limited per event Frontend Frontend Frontend Frontend Fro...
Dispatcher Performance Frontend Frontend Frontend Frontend Frontend Dispatcher ? events/second 151
5K We can publish 50K likes per second with just 10 machines 152
LinkedIn Realtime Performance >100K Persistent Connections/machine Dispatcher Publish QPS/machine 75ms E2E Publish Latency...
E2E Publish Latency Frontend Dispatcher t1 Key-Value Store 154
E2E Publish Latency Frontend Dispatcher t1 Key-Value Store t2 t2-t1 = 75ms p90 155
LinkedIn Realtime Performance >100K Persistent Connections/machine Dispatcher Publish QPS/machine 75ms E2E Publish Latency...
LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinlatency Measurement System E2E Publish Latency 157
Why does this system scale? Horizontally Scalable System Frontend Dispatcher Frontend Frontend Frontend Dispatcher Dispatc...
Active Status/Presence Online/Offline indicators LINKEDIN REALTIME TECHNOLOGY 159
LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinpresence Active Status/Presence 160
Key takeaways 161
R E A LT I M E I N T E R A C T I O N S Enable dynamic instant experiences on your apps 162
E V E N T S O U R C E / S S E Built-in support in most server frameworks Readily available client libraries HTTP/1.1 200 O...
P l a y F r a m e w o r k a n d A k k a A c t o r s Powerful frameworks to manage millions of connections State BehaviorMa...
D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 165
S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 166
R E A LT I M E I N T E R A C T I O N P L A T F O R M Can be built on any server/storage technology Horizontally scalable F...
R E A LT I M E I N T E R A C T I O N P L A T F O R M You can do it for your app! 168
R E A LT I M E E N G I N E E R , L I N K E D I N @agupta03 t i n y . c c / q c o n 2 0 2 0 169 AMA at 1.40pm at Guild, Mez...
Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ linkedin-play-akka-distribut...
Upcoming SlideShare
Loading in …5
×

Streaming a Million Likes/Second: Real-Time Interactions on Live Video

1,423 views

Published on

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/39NIjLV.

Akhilesh Gupta does a technical deep-dive into how Linkedin uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers. Filmed at qconlondon.com.

Akhilesh Gupta is the technical lead for LinkedIn's Real-time delivery infrastructure and LinkedIn Messaging. He has been working on the revamp of LinkedIn’s offerings to instant, real-time experiences. Before this, he was the head of engineering for the Ride Experience program at Uber Technologies in San Francisco.

Published in: Technology
no profile picture user

  • Login to see the comments

Streaming a Million Likes/Second: Real-Time Interactions on Live Video

  1. 1. Streaming a million likes/second Real-time interactions on Live Video Akhilesh Gupta Realtime Engineer, LinkedIn 2
  2. 2. InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ linkedin-play-akka-distributed- systems/ • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
  4. 4. tiny.cc/qconlive 3 Login with LinkedIn credentials
  5. 5. Which was the largest live stream in the world? (largest number of concurrent viewers) ? 4
  6. 6. INDIA VS NEW ZEALAND WORLD CUP Largest Live Stream >25.3M viewers © The Indian Express https://images.indianexpress.com/2019/06/13th-june_india-vs-new-zealand_759.jpg 5
  7. 7. 6
  8. 8. D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 7
  9. 9. 8
  10. 10. What is a Live Video? 9
  11. 11. What are real-time interactions on Live Video? 10
  12. 12. Mobile LINKEDIN LIVE Likes/Comments Concurrent Viewer Counts 11
  13. 13. LINKEDIN LIVE Desktop Likes/Comments Concurrent Viewer Counts 12
  14. 14. Distribution of Likes A S 13
  15. 15. Distribution of Likes A S Simple HTTP Request 14
  16. 16. Distribution of Likes A S Simple HTTP Request ? 15
  17. 17. Challenge 1:The delivery pipe 16
  18. 18. The delivery pipe Real-time Delivery A S Likes Backend (Publisher) 17
  19. 19. The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Simple HTTP Request 18
  20. 20. The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Persistent Connection 19
  21. 21. The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Persistent Connection 20
  22. 22. Persistent Connection Real-time Delivery 21
  23. 23. TRY IT! Login at linkedin.com Persistent Connection tiny.cc/realtime 22
  24. 24. 23
  25. 25. EVENTSOURCE INTERFACE CLIENT REQUEST HTTP Long Poll with Server Sent Events GET /connect HTTP/1.1 Accept: text/event-stream 24
  26. 26. EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream 25
  27. 27. EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {“like” object} 26
  28. 28. EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {“like” object} .. data: {“comment” object} 27
  29. 29. EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {“like” object} .. data: {“comment” object} .. data: {“like” object} 28
  30. 30. WEB EventSource Client Libraries var evtSource = new EventSource("https://www.linkedin.com/realtime/connect"); evtSource.onmessage = function(e) { var likeObject = JSON.parse(e.data); } 29
  31. 31. ANDROID & IOS SUPPORT EventSource Client Libraries Android: https://github.com/tylerjroach/eventsource-android iOS Swift: https://github.com/inaka/EventSource 30
  32. 32. Next Challenge? Real-time Delivery 31
  33. 33. Next Challenge: Multiple Connections Real-time Delivery 32
  34. 34. Challenge 2: Connection Management 33
  35. 35. PLAY & AKKA Connection Management Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications 34
  36. 36. © Disney, All Rights Reserved, Pirates of the Caribbean 35
  37. 37. AN AKKA ACTOR Connection Management State Behavior 36
  38. 38. AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor 37
  39. 39. AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor 38
  40. 40. AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor Millions of Actors 39
  41. 41. AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor No. of threads proportional to no. of cores Millions of Actors 40
  42. 42. AN AKKA ACTOR Connection Management Connection Publish Event ABC Events to be published Millions of Connections 41
  43. 43. INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned connectionIdA public Result listen() { } 42
  44. 44. INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned connectionIdA public Result listen() { return ok(EventSource.whenConnected(eventSource -> { String connectionId = UUID.randomUUID().toString(); })); } 43
  45. 45. INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned connectionIdA public Result listen() { return ok(EventSource.whenConnected(eventSource -> { String connectionId = UUID.randomUUID().toString(); // construct an Akka Actor to manage connection _actorSystem.actorOf( ClientConnectionActor.props(connectionId, eventSource), connectionId ); })); } 44
  46. 46. Publish Events Using Akka Actors Akka Actor Real-time Delivery System Connection ID cid1 45
  47. 47. EVENT PUBLISH FROM PUBLISHER Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 46
  48. 48. EVENT PUBLISH FROM SUPERVISOR TO CHILD ACTORS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 47
  49. 49. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 48
  50. 50. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publishing on EventSource eventSource.send(<like object>); 49
  51. 51. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 50
  52. 52. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 51
  53. 53. EVENTSOURCE INTERFACE Received Events on the Clients HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {“User A liked the video…” object} 52
  54. 54. Next Challenge? Real-time Delivery 53
  55. 55. Next Challenge: Multiple Live Videos Real-time Delivery 54
  56. 56. Challenge 3:Multiple Live Videos 55
  57. 57. START WATCHING LIVE VIDEO Multiple Live Videos cid3 cid5 Live Video 1 Live Video 2 Real-time Delivery System 56
  58. 58. HTTP SUBSCRIPTION REQUEST Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid3 cid5 Live Video 1 Live Video 2 /subscribe 57
  59. 59. IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid5 Live Video 2 /subscribe /subscribe cid3Live Video 1 58
  60. 60. IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid3 cid5 Live Video 1 Live Video 2 /subscribe /subscribe 59
  61. 61. Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 60
  62. 62. PUBLISHER PUBLISH TO SUPERVISOR ACTOR Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 61
  63. 63. SUBSCRIBER LOOKUP FROM IN MEMORY TABLE Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 62
  64. 64. PUBLISH TO CLIENT CONNECTIONS Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 63
  65. 65. PUBLISHER PUBLISH TO SUPERVISOR ACTOR Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 64
  66. 66. SUBSCRIBER LOOKUP FROM IN MEMORY TABLE Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 65
  67. 67. PUBLISH TO CLIENT CONNECTIONS Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 66
  68. 68. Next Challenge? Real-time Delivery 67
  69. 69. Next Challenge: 10K Concurrent Viewers Real-time Delivery 68
  70. 70. Challenge 4:10K Concurrent Viewers 69
  71. 71. 70
  72. 72. 10K Concurrent Viewers Real-time Delivery 71
  73. 73. S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 72
  74. 74. 10K Concurrent Viewers Real-time Delivery 73
  75. 75. 10K Concurrent Viewers Frontend Frontend 74
  76. 76. 10K Concurrent Viewers Frontend Frontend 75
  77. 77. DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 76
  78. 78. DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 77
  79. 79. DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 78
  80. 80. Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Real-time Dispatcher 79 Live Video 1
  81. 81. HTTP SUBSCRIPTION REQUEST Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe 80 Live Video 1
  82. 82. SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe /subscribe 81 Live Video 1
  83. 83. SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe /subscribe 82 node2 Live Video 1
  84. 84. SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe /subscribe 83 node2 Live Video 1
  85. 85. Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2 84
  86. 86. PUBLISHER PUBLISH TO DISPATCHER Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Dispatcher 85 Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2
  87. 87. SUBSCRIBER LOOKUP Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Dispatcher 86 Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2
  88. 88. PUBLISH TO SUBSCRIBED FRONTEND NODES Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Dispatcher 87 Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2
  89. 89. Frontend Node to Client Publish Frontend 88
  90. 90. Frontend Node to Client Publish Frontend 89
  91. 91. What’s the next challenge? Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 90
  92. 92. Challenge 5:100 Likes/second 91
  93. 93. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 92
  94. 94. S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 93
  95. 95. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 94
  96. 96. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Dispatcher Frontend Frontend 95
  97. 97. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Dispatcher Frontend Frontend 96
  98. 98. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Dispatcher Frontend Frontend 97
  99. 99. COUCHBASE/REDIS ETC. Key-Value Store for Subscriptions Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Key-Value Store Dispatcher 98
  100. 100. Publish to Dispatcher Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 99
  101. 101. Subscriber lookup from KV Store Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 100
  102. 102. Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 101
  103. 103. Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher node1 node2 node3 node4 node5 node6 node7 Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 102
  104. 104. Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher node1 node2 node3 node4 node5 node6 node7 Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 103
  105. 105. Challenge 6: 100 Likes/s, 10K Viewers Distribution of 1M Likes/s 104
  106. 106. Mobile LINKEDIN LIVE Likes 105
  107. 107. The Subscription Flow Frontend Dispatcher Key-Value Store 1 106
  108. 108. START WATCHING LIVE VIDEO Viewers subscribing to Live Video Likes cid3 cid5 Live Video 1 Live Video 2 Frontend 107
  109. 109. HTTP SUBSCRIPTION REQUEST Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /subscribe Frontend In-Memory Subscriptions Table 108
  110. 110. FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid5 Live Video 2 /subscribe /subscribe cid3Live Video 1 Frontend In-Memory Subscriptions Table 109
  111. 111. FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /subscribe /subscribe Frontend In-Memory Subscriptions Table 110
  112. 112. FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /subscribe /subscribe Frontend In-Memory Subscriptions Table 111
  113. 113. The Subscription Flow Frontend Dispatcher Key-Value Store 1 2 3 112
  114. 114. FRONTEND RECEIVES SUBSCRIPTION REQUEST Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Key-Value Store Dispatcher Dispatcher 113
  115. 115. FRONTEND HTTP SUBSCRIPTION REQUEST TO DISPATCHER Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host /subscribe Key-Value Store Dispatcher Dispatcher 114
  116. 116. DISPATCHER SUBSCRIPTION ENTRY IN KV STORE Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host /subscribe /subscribe Dispatcher Dispatcher Key-Value Store 115
  117. 117. DISPATCHER SUBSCRIPTION ENTRY IN KV STORE Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host /subscribe /subscribe Dispatcher Dispatcher Key-Value Store 116
  118. 118. The Publish Flow Frontend Dispatcher Key-Value Store 1 117
  119. 119. Viewers liking Live Videos Likes Backend (Publisher) 118
  120. 120. Likes Published to Backend Likes Backend (Publisher) 119
  121. 121. The Publish Flow Frontend Dispatcher Key-Value Store 1 2 345 120
  122. 122. The Publish Flow Frontend Dispatcher Key-Value Store 2 345 121
  123. 123. BACKEND PUBLISH TO DISPATCHER NODES Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 122
  124. 124. SUBSCRIBER LOOKUP FROM KV STORE Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 123
  125. 125. SUBSCRIBER LOOKUP FROM KV STORE Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 124
  126. 126. PUBLISH TO FRONTEND NODES Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher node1 node2 node3 node4 node5 node6 node7 Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 125
  127. 127. The Publish Flow Frontend Dispatcher Key-Value Store 5 126
  128. 128. DISPATCHER PUBLISH TO FRONTEND NODE SUPERVISOR ACTOR Frontend Publish to Mobile Clients Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 Dispatcher Frontend In-Memory Subscriptions Table 127
  129. 129. SUBSCRIBED CLIENT CONNECTION LOOKUP FROM IN-MEMORY TABLE Frontend Publish to Mobile Clients Actor 1 Frontend Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 Dispatcher In-Memory Subscriptions Table 128
  130. 130. PUBLISH TO MOBILE CLIENTS Frontend Publish to Mobile Clients Actor 1 Frontend Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 Dispatcher In-Memory Subscriptions Table 129
  131. 131. Bonus Challenge: Another data center 130
  132. 132. 131
  133. 133. D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 132
  134. 134. Frontend DispatcherDC-1 133
  135. 135. Frontend Dispatcher Frontend DispatcherDC-2 DC-1 134
  136. 136. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 135
  137. 137. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 136
  138. 138. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 137
  139. 139. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 138
  140. 140. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 CrossDataCenterPublish 139
  141. 141. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 140
  142. 142. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 141
  143. 143. Performance & Scale 142
  144. 144. How many persistent connections can a single machine hold? Frontend Server ? 143
  145. 145. > 100,000 144
  146. 146. ROYAL WEDDING Second Largest Stream >18M viewers © time.com 145
  147. 147. > 100,000 We can hold those 18M connections with just 180 machines 146
  148. 148. LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinscaling >100K 147
  149. 149. LinkedIn Realtime Performance Persistent Connections/machine 5K/s Dispatcher Publish QPS/machine 75ms E2E Publish Latency >100K 148
  150. 150. Dispatcher Performance Frontend Frontend Frontend Frontend Frontend Dispatcher How many events per second can be published to a single dispatcher machine? ? events/second 149
  151. 151. Dispatcher Performance Number of frontend nodes to publish to is limited per event Frontend Frontend Frontend Frontend Frontend Dispatcher ? events/second 150
  152. 152. Dispatcher Performance Frontend Frontend Frontend Frontend Frontend Dispatcher ? events/second 151
  153. 153. 5K We can publish 50K likes per second with just 10 machines 152
  154. 154. LinkedIn Realtime Performance >100K Persistent Connections/machine Dispatcher Publish QPS/machine 75ms E2E Publish Latency 5K 153
  155. 155. E2E Publish Latency Frontend Dispatcher t1 Key-Value Store 154
  156. 156. E2E Publish Latency Frontend Dispatcher t1 Key-Value Store t2 t2-t1 = 75ms p90 155
  157. 157. LinkedIn Realtime Performance >100K Persistent Connections/machine Dispatcher Publish QPS/machine 75ms E2E Publish Latency 5K 156
  158. 158. LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinlatency Measurement System E2E Publish Latency 157
  159. 159. Why does this system scale? Horizontally Scalable System Frontend Dispatcher Frontend Frontend Frontend Dispatcher Dispatcher Dispatcher Distributed K/V Storage System 158
  160. 160. Active Status/Presence Online/Offline indicators LINKEDIN REALTIME TECHNOLOGY 159
  161. 161. LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinpresence Active Status/Presence 160
  162. 162. Key takeaways 161
  163. 163. R E A LT I M E I N T E R A C T I O N S Enable dynamic instant experiences on your apps 162
  164. 164. E V E N T S O U R C E / S S E Built-in support in most server frameworks Readily available client libraries HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream 163
  165. 165. P l a y F r a m e w o r k a n d A k k a A c t o r s Powerful frameworks to manage millions of connections State BehaviorMailbox 164
  166. 166. D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 165
  167. 167. S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 166
  168. 168. R E A LT I M E I N T E R A C T I O N P L A T F O R M Can be built on any server/storage technology Horizontally scalable Frontend Dispatcher Key- 167
  169. 169. R E A LT I M E I N T E R A C T I O N P L A T F O R M You can do it for your app! 168
  170. 170. R E A LT I M E E N G I N E E R , L I N K E D I N @agupta03 t i n y . c c / q c o n 2 0 2 0 169 AMA at 1.40pm at Guild, Mezzanine
  171. 171. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ linkedin-play-akka-distributed-systems/

×