Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - re:Invent 2017
•
6 likes•2,896 views
Over 100 million subscribers from over 190 countries enjoy the Netflix service. This leads to over a trillion events, amounting to 3 PB, flowing through the Keystone infrastructure to help improve customer experience and glean business insights. The self-serve Keystone stream processing service processes these messages in near real-time with at-least once semantics in the cloud. This enables the users to focus on extracting insights, and not worry about building out scalable infrastructure. In this session, I share the benefits and our experience building the platform.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - re:Invent 2017
Similar to Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - re:Invent 2017 (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - re:Invent 2017
- 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Netflix Keystone SPaaS S t r e a m P r o c e s s i n g A s a S e r v i c e A B D 3 2 0 Monal Daxini @monaldax #reInvent #Netflix Stream Processing Infrastructure
- 2. @monaldax ● Data Engineer Why stream processing, and what to expect from a platform? ● Data Leader Product / Vision of Stream Processing As a Service platform ● Platform engineer How to build and operate a a Stream What Do I Get Out Of This Talk? Different perspectives: @monaldax
- 3. @monaldax ● I will focus on stream processing platform for business insights, which my team builds mostly based on Flink ● I won’t be addressing operational insights for which we have different systems
- 6. @monaldax ● Low latency insights and analytics ● Processing data as it arrives helps spread workload over time, & reduce processing redundancy ● Need to process unbounded data sets becoming increasingly common Why Real Time Data?
- 7. @monaldax ● Enable users to focus on data and business insights, and not worry about building stream processing infrastructure and tooling Why Build A Stream Processing Platform?
- 8. @monaldax What Does A Stream Processing Platform Offer?
- 9. @monaldax Platform Needs To Offer Robust Way To Process Streams Allowing To Tradeoff Between Ease, Capability, &Flexibility SPaa S
- 10. @monaldax Point & Click Routing, Filtering, Projection Streaming Jobs ● Support Streaming SQL Future ● Interactive exploration of streams for quick prototyping Future Stream Processing as a Service platform offers
- 11. @monaldax Point & Click Routing, Filtering, Projection
- 12. @monaldax Event Producers Sinks Ingest Pipelines Are The Backbone Of A Real-time Data Infrastructure SERVERL ESS Turnkey 100% in AWS
- 13. @monaldax Keystone Pipeline– Provision A Managed Data Stream 📽
- 14. @monaldax Keystone Self-serve - Optional Filter & Message Parser * We would eventually like to move away from xpath & our custom parser
- 15. @monaldax * We would eventually like to move away from xpath & our custom parser Keystone Self-serve – Message Format
- 16. @monaldax Keystone Self-Serve – Optional Projection 📽
- 17. @monaldax Keystone Self-serve – Elasticsearch Sink Config
- 18. @monaldax Keystone Self-serve – Kafka Sink Partition Key Support
- 19. @monaldax Keystone - Configure 1 Data Stream, A Filter, & 3 Sinks
- 20. Event Producer Create Kafka Topic, And Three Separate Jobs SPaaS Router Fronting Kafka KSGateway Consumer Kafka KC W Elasticsearc h 3 Jobs1 Topic Keystone Management 1 Topic @monaldax
- 21. Event Flow: Producer Uses Kafka Client Wrapper Or Proxy SPaaS Router Fronting Kafka Event Producer KSGateway Consumer Kafka Keystone Management KC W Elasticsearc h @monaldax
- 22. Event Flow: Events Queued In Kafka SPaaS Router Fronting Kafka Event Producer KSGateway Consumer Kafka KC W Elasticsearc h 3 instance s Keystone Management @monaldax
- 23. Event Flow: Each Router Reads From Source, Optionally Applies Filter & Projection SPaaS Router Fronting Kafka Event Producer KSGateway Consumer Kafka KC W Elasticsearc h 3 instance s Keystone Management @monaldax
- 24. Event Flow: Each Router Writes To Their Respective Sinks SPaaS Router Fronting Kafka Event Producer KSGateway Consumer Kafka KC W Elasticsearc h 3 instance s Non- Keyed Keyed Supported Keystone Management @monaldax
- 25. Dashboard & Alert Config Generation For Provisioned Streams
- 26. Searchable Router Job Logs
- 27. Flink Job Web UI @monaldax k
- 29. Data Stream Operations is Managed • Fully managed scaling • Managed capacity planning • 24 X 7 availability [Scale] • Garbage collect unused streams @monaldax
- 30. Keystone Pipeline - The Road Ahead • Additional components – UDFs, Data Hygiene, Data Alerting, etc • Component chaining • Schema Support • Data Lineage • Cost attribution @monaldax
- 31. @monaldax Point & Click Routing, Filtering, Projection (prod) Streaming Jobs
- 32. Why A Streaming Job? • When we need more flexibility and power than what Point & Click pipeline offers, use stream processing jobs. @monaldax
- 33. Generate Streaming Job From Template @monaldax
- 35. Run And Debug Locally In The IDE @monaldax
- 36. Create A New Streaming Job Config For Deployment @monaldax
- 37. Deploying A Streaming Job In Test @monaldax
- 38. Deploying A Streaming Job In Other Environments @monaldax
- 39. Deployment Status Of A Sample Streaming Job
- 40. Streaming Job Actions & Links @monaldax
- 43. @monaldax ● Use case specific consulting ● Recipes ● Examples and Documentation In Addition, Consulting & Documentation
- 45. Broadly, Two Categories Of Streaming Jobs • Stateless • No state maintained across events • Stateful • State maintained across events @monaldax
- 46. Event Producer Streaming Job In Context Of Keystone Pipeline SPaaS Router Fronting Kafka KSGateway Consumer Kafka Keystone Management KC W Elasticsearc h Streaming Job @monaldax
- 47. Image adapted from: Stateless Stream Processor – No Internal State @monaldax
- 48. Stateless Stream Processor – External State Image adapted from: Stephan Ewen@monaldax
- 49. Stateless Example: Generating Plays Feed For Personalization, And Discovery Of Shows
- 50. @monaldax Stateless Streaming Job Use Case: High Level Architecture Enriching And Identifying Certain Plays Playback History Service Video Metada ta Streaming Job Play Logs Live Service Lookup Data
- 51. Stateful Stream Processing Image adapted from: Stephan Ewen @monaldax
- 52. Stateful Example: Creating Search Sessions
- 53. Search Personalization – Custom Windowing On Out-of-order Events ...... S ES ……….Session 2: S Hour s S E Session 1: SE … @monaldax
- 54. Streaming Application Flink Engine Local State Stateful Streaming Application With Local State, Checkpoints, And Savepoints Sinks Savepoints (Explicitly Triggered) Checkpoi nts (Automatic) Sources @monaldax
- 55. Streaming Job (Flink) Savepoint Tooling Support • S3 based multi-tenant storage management • Auto savepoint and resume from savepoint on redeploy • Resume from an existing savepoint @monaldax
- 56. Streaming Job (Flink) High Level Features • Stateless jobs • Event enrichment support by accessing services using platform thick clients • Stateful jobs 100s of GB, with larger state support in the works • Reusable blocks (in progress) • Job development, deployment, and monitoring tooling (alpha)@monaldax
- 57. Streaming Jobs - The Road Ahead • Easy resource provisioning estimates • Flink support for reading and writing from data warehouse, backfill • Continue to evolve tooling and support for large state • Reusable Components - sources, sinks, operators, schema support, data hygiene • Tooling support for Spark Streaming @monaldax
- 58. @monaldax Scale?
- 59. Prod – Trending Events & Scale With Events Flowing To Hive, Elasticsearch, Kafka ≅ 80B to 1.3T • 1.3T+ events processed per day • 600B to 1T unique per day • 2+ PB in 4.5+ PB out per day • Peak: 12M events in / sec & 36 GB / sec @monaldax
- 60. @monaldax Keystone Router Stream Processing Jobs Scale m4.4xl
- 61. @monaldax How Do We Do It?
- 62. @monaldax RTDI Consists Of 4 Systems. Keystone Pipeline Runs 24 X 7, & Does Not Impact Members Ability To Play Videos Keystone Stream Processing (SPaaS) Keystone Management Keystone Messaging 24 x 7 - Dev - Test - Prod Granular shadowin g
- 63. Event Producer Components & Streaming Jobs SPaaS Router Fronting Kafka KSGateway Consumer Kafka Keystone Management KC W Hive Elasticsearc h Streaming Job @monaldax
- 65. • Inject event metadata - GUID, timestamp, host, app • Transparent and dynamic traffic routing for producers • Chaski - Custom binary data wrapper within Keystone pipeline • Multiple serialization support & Additional metadata Producer Library - Kafka Client Wrapper @monaldax
- 66. Streaming JobEvent Producer Boundary Of Custom Binary Data Wrapper SPaaS Router Fronting Kafka KSGateway Consumer Kafka Keystone Management KC W Hive Elasticsearc h @monaldax
- 67. • Automated Kafka producer buffer (60s) tuning based on traffic • Best effort delivery, Prioritizes host application availability • acks=1, Do not block to send events, Unclean leader election • Non-keyed messages, retry send to available Producer Library - Kafka Client Wrapper @monaldax
- 68. Event Producer Ksgateway - Event Proxy For Non-java Clients, REST & GRPC SPaaS Router Fronting Kafka KSGateway Consumer Kafka Keystone Management KC W Hive Elasticsearc h Streaming Job @monaldax
- 69. Event Producer Kafka Clusters (0.10) on Amazon EC2 SPaaS Router Fronting Kafka KSGateway Consumer Kafka Keystone Management KC W Elasticsearc h Streaming Job @monaldax
- 70. • Have message sizes > 1MB and up to 10MB • Large Scale Keystone Ingest pipelines results in large fan out • Lower Latency – used for ad-hoc messaging as well • Open source – enhance, patch, or extend Why Kafka? @monaldax
- 71. Scale for Large Fan-out and Isolation - Cascading Topology Fronting Kafka Consumer Kafka Consumer @monaldax
- 72. Alternative: Logical Stream (Topic) Spread Across Multiple Topics Across Multiple Clusters (WIP) Multi-Cluster Producer Multi- Cluster Consumer @monaldax
- 73. • Dedicated Zookeeper cluster per Kafka cluster • Small Clusters < 200 brokers, partitions <= 10K • Partitions distributed evenly across brokers • Rack-aware replica assignment, brokers spread in 3 Zones • 2 copies & Unclean leader election on • Non-transactional Kafka Deployment Strategies – Version 0.10 (YMMV) @monaldax
- 74. • 36+ Kafka & Zookeeper clusters • 4000+ brokers (EC2), 700+ topics • 3000+ d2.xl, 900+ i2.2xl • Highly available 99.99%+ • Retention 2hr, 4hr, 8hr, 24hr Kafka Clusters Scale @monaldax
- 76. High-level Stream Processing Platform Architecture - Routers Keystone Management Point & Click Router Streaming Job Container Runtime 1. Create Streaming Job 2. Launch Job with Config, Source, Sink, Filters, Projections, etc. 3. Launch Containers • Immutable Image • Automated, system driven config overrides @monaldax
- 77. • Keystone pipeline is built on Flink Routers • Each Flink Router is a stream processing job • Router provisioning based on incoming traffic or estimates • Runs on containers atop EC2 • Island mode - single AWS Region Streaming Jobs 1.3.2 @monaldax
- 78. High-level Stream Processing Platform Architecture Streaming Jobs Keystone Management Point & Click or Streaming Job Container Runtime 1. Create Streaming Job 2. Launch Job with Config overrides 3. Launch Containers • Immutable Image • User driven config overrides @monaldax
- 79. Stream Processing Platform - Layered cake Amazon EC2 Titus Container Runtime Stream Processing Platform (Flink Streaming Engine, Config Management) Reusable Components Source & Sink Connectors, Filtering, Projection, etc. Routers (Streaming Job) Streaming Jobs @monaldax
- 80. @monaldax Flink Job Cluster In HA Mode Zookeeper Job Manager Leader (WebUI) Task Manager Task Manager Task Manager Job Manager (WebUI) One dedicated Zookeeper cluster for all streamig Jobs
- 81. Flink Task Slots & Automatic Operator Chaining Image: Flink 1.2 documentation@monaldax
- 82. @monaldax Flink Job Cluster In HA Mode With Checkpoints Zookeeper Job Manager (Leader) Task Manager Task Manager Task Manager Job Manager State Checkpoints State Metadata Checkpoint s
- 83. Flink Checkpoints Similar To 2 Phase Commit Image: Flink 1.2 documentation@monaldax
- 84. @monaldax Titus Job Task Manager I P Titus Host 4 Titus Host 5 Checkpoints Are Taken Often Zookeep er Job Manager (standby) Job Manager (master) Task Manager Titus Host 1 I P Titus Host 1 …. Task Manager Titus Host 2 I P Titus Job I P I P AWS VPC State - Checkpoints - Kafka Offset Save
- 85. @monaldax Titus Job Task Manager I P Titus Host 4 Titus Host 5 Checkpoints Are Taken Often. A Container Could Fail… Zookeep er Job Manager (standby) Job Manager (master) Task Manager Titus Host 1 I P Titus Host 1 …. Task Manager Titus Host 2 I P Titus Job I P I P AWS VPC State - Checkpoints - Kafka Offset Save X
- 86. @monaldax Titus Job Task Manager I P Titus Host 4 Titus Host 5 Zookeep er Job Manager (standby) Job Manager (master) Task Manager Titus Host 1 I P Titus Host 2 …. Task Manager Titus Host 3 I P Titus Job I P I P AWS VPC State - Checkpoints - Kafka Offset Restor e Restored To Last Checkpoint, Partially Recovery Supported Replacement container
- 87. Event Producer and Streaming Jobs Management SPaaS Router Fronting Kafka KSGateway Consumer Kafka Keystone Management KC W Hive Elasticsearc h Streaming Job @monaldax
- 88. @monaldax Keystone Management Current Architecture - Imperative Composab le Joblets Composab le Joblets
- 89. @monaldax Keystone Management New Architecture (WIP) – Declarative
- 90. @monaldax Keystone Management New Architecture (WIP)
- 91. • The ability to pass data along the chain of Joblets within a Job • Locks and semaphores on resources spanning across jobs • Customization and integration into Netflix ecosystem – Eureka, etc. Keystone Management Unique Features @monaldax
- 92. @monaldax How Do We Operate It? Scale Operations Using Systems Not Humans
- 93. • No separate Ops team • No separate QA team • No separate Dev team • It’s all done by developers of the Real Time Data Infrastructure We Run What We Build! @monaldax
- 94. • We rely on metrics, monitoring, alerting & paging, & automation • Separate metrics system – Atlas • Separate alert configuration and alert actions system • Options for separate system to run cross-system automation tasks We Leverage Other Netflix Systems @monaldax
- 95. Easy Alert Configuration And Status @monaldax
- 96. Easy View Of Fired Alerts @monaldax
- 97. Streaming JobEvent Producer Operating Ksgateway - Event Proxy For Non- Java Clients SPaaS Router Fronting Kafka KSGateway Consumer Kafka Keystone Management Hive Elasticsearc h • Stateless Service • Scaled Using Elastic Load Balancing and Auto Scaling Group • Pre-scaled for planned increase in traffic @monaldax
- 98. Streaming JobEvent Producer Event Producer Related Monitoring And Alerts SPaaS Router Fronting Kafka KSGateway Consumer Kafka Keystone Management KC W Elasticsearc h @monaldax
- 99. @monaldax Monitoring Producer, Alert On Drop Rate
- 101. @monaldax Kafka Failover - Fronting Kafka Clusters
- 102. @monaldax Fully Automated Kafka Cluster Failover – As Fast As 5 Minute
- 103. @monaldax Kafka Cluster & Routers In Healthy State Flink RouterFronting Kafka Event Producer
- 104. @monaldax Issue With Kafka Cluster Flink RouterFronting Kafka Event Producer X
- 105. @monaldax Launch Backup Kafka Cluster With Same Number Of Instances, But Smaller Instance Type Flink RouterFronting Kafka Event Producer Bring up failover Kafka cluster Copy metadata from Zookeeper X
- 106. @monaldax Change Producer Config To Produce To Failover Cluster, And Launch Routers For Failover Traffic Flink RouterFronting Kafka Event Producer Failover Flink Router X
- 107. @monaldax Change Producer Config To Original Cluster, And Finish Draining Events From Backup Flink Router Flink RouterFronting Kafka Event Producer Failover Flink Router
- 108. @monaldax Decommission Backup Cluster And Router Once Original Cluster Is Fixed, Or A Replacement Cluster Is Live Flink RouterFronting Kafka Event Producer Failover Flink Router X X
- 109. @monaldax Flink RouterFronting Kafka Event Producer Back To Steady State With Click Of A Button
- 110. • Failover currently supported for Fronting Kafka clusters only • We are working on multi-consumer client with support for keyed message to support failover of consumer Kafka clusters. Consumer Kafka Clusters @monaldax
- 111. Planned & Regular Kafka Kong This Automation Also Serves As Kafka Kong, A Tool That Follows Principles Of Chaos Engineering @monaldax
- 112. • Over provision for variations and traffic for failover • Broker health & outlier detection and auto termination • 99 percentile response time • Broker TCP timeouts, errors, retransmissions • Producer’s send latency Kafka Operation Strategies (YMMV) @monaldax
- 113. • Scale up by • Adding partitions – to new brokers, requires no keyed messages • Partition reassignment – in small batches with custom tool • Scale down by • Create New topics / New clusters • Create new clusters - use Kafka failover Kafka Operation Strategies (YMMV) @monaldax
- 114. Event Producer Stream Processing Platform Router Fronting Kafka KSGateway Consumer Kafka Keystone Management KC W Elasticsearc h Flink Streaming Job @monaldax
- 115. • Container replacement • Checkpoints and Savepoints • Keep retrying if event data format is valid • Isolation – issue with one sink does not impact another Routers & Streaming Job Fault Tolerance By Design @monaldax
- 116. • Provision new or updated streams • Bulk updates and terminate routers and re- deployment • Automatic partial recovery allows zero-touch migration of underlying container infrastructure • Manual – KSRunbook Router Deployment Automation @monaldax
- 117. Manual Intervention, We Have Runbook. Goal Is To Automate And Keep Runbook Small @monaldax
- 118. • Per stream provisioning based on past weeks traffic or bit rate estimate • Provision buffer capacity • Run 1 additional container for latency sensitive consumers • Manual, % increase, easy to compute and deploy • Plan capacity to handle service failover, and Router Capacity Planning And Provisioning @monaldax
- 119. Admin Tooling To Scale Up Manually, Or To Deploy A New Build @monaldax
- 120. Application Metrics – Router Message Flow @monaldax
- 121. Application Metrics – Router filtering @monaldax
- 122. Platform-level Metrics – Kafka Offset Metrics
- 123. System Metrics - Router JVM Metrics @monaldax
- 124. Alerts– Hive Sink Router @monaldax
- 125. @monaldax Flink Streaming Job ● Split between application and infrastructure ● Metrics and monitoring and ● Alerts ● Paging and on-call rotations ● Platform customers follow the same “We build it we run it model”
- 126. Example Streaming Job Application Level Simulated Metrics
- 127. Example Streaming Job System Level Simulated Metrics
- 128. @monaldax Operations – The road ahead ● True auto scaling ● Bootstrap capacity planning for stateful streaming jobs ● Automated Canary tooling & Data parity ● Point and Click components quick testing, and performance profiling ● E.g., - iterating over a Filter definition
- 129. @monaldax I Want To Learn More ● http://bit.ly/mLOOP - Deep dive into Unbounded Data Processing Systems ● http://bit.ly/m17FF - Keynote – Stream Processing with Flink at Netflix ● http://bit.ly/2BoYAq0 - Multi-tenant Multi-cluster Kafka
- 130. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. THANK YOU! M o n a l D a x i n i @ m o n a l d a x Que sti o ns?