Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Netflix Conductor
A Microservices Orchestrator
Viren Baraiya, Vikram Singh
Content Platform Engineering - CPE
〉 Studio In the Cloud
〉 Content Ingest from Studio Partners
〉 Title Setup - Making it l...
CPE - Processes (some of many)
〉 Content Ingest & Delivery
〉 Title Setup
〉 IMF*
Deliveries
〉 Encodes and Deployments
〉 Con...
Once Upon A Time ...
〉 Peer to Peer Messaging
〉 10’s MM messages per day
〉 Process flows embedded in applications
〉 Lack o...
Example
Peer to Peer
Application C Application BApplication BApplication A
Request Content Content Inspection Result Encode Publis...
Peer to Peer
Application C Application BApplication BApplication A
Request Content Content Inspection Result Encode Publis...
〉 Orchestration Engine
〉 Orchestration Engine
〉 Open Source (Apache 2.0)
Conductor - Design Goals
〉 BYO Task (Reuse existing code)
〉 REST/HTTP support
〉 Extensible and Hackable
〉 JSON based DSL t...
Same Flow - New Flavor
Request
Content
Content
Inspection
Result Encode PublishStart
Stop
Conductor
Application A
Task
Req...
Architecture
API
Workflows Metadata Tasks
SERVICE
Workflow Service Task Service
Decider Service Queue Service
STORE
Storag...
Scaling up Conductor
〉 Peer-to-Peer - Scale horizontally
〉 Stateless server - state is persisted in database
〉 Storage sca...
Storage
Dynomite
〉 Generic Dynamo implementation (Redis, Memcache)
〉 Multi-datacenter
〉 Highly available
〉 Peer-to-Peer
El...
Dyno-Queues
〉 Distributed lock free queues used by Conductor
〉 OSS
〉 Apache 2.0 License
〉 https://github.com/Netflix/dyno-...
Task
〉 Reusable stateless components
〉 System Tasks (Fork / Join, Decision, etc.)
〉 Remote Workers (Java, Python, lang. ag...
Concepts
Task
〉 Reusable stateless components
〉 System Tasks (Fork / Join, Decision, etc.)
〉 Remote Workers (Java, Python,...
Tasks Definition
〉 Retries
〉 Timeouts
〉 Documentation for Input / Output
〉 Registry
Task Definition - Example
{
"name": "encode_task",
"retryCount": 2,
"timeoutSeconds": 3600,
"inputKeys": [
"fileLocation",...
Workflow Definition
〉 Logical Flow of Tasks
〉 Error / Failure Handling
〉 Input / Output transformation
〉 JSONPath Based
〉 ...
Workflow - Example
{
"name": "encode_and_deploy",
"description": "Encodes a file and deploys to CDN",
"version": 1,
"tasks...
Input / Output
〉 Input to tasks are transformed
〉 Refer Input / Outputs from
〉 Task
〉 Workflow
〉 Complex JSON transformati...
Example
{
"inputParameters": {
"movieId": "${workflow.input.movieId}",
"url": "${workflow.input.fileLocation}",
"lang": "$...
Example
{
"inputParameters": {
"movieId": "${workflow.input.movieId}",
"url": "${workflow.input.fileLocation}",
"lang": "$...
Example
{
"inputParameters": {
"movieId": "${workflow.input.movieId}",
"url": "${workflow.input.fileLocation}",
"lang": "$...
Execution Flow
Putting It All Together
Execution Flow
App A
Conductor
App A
Task
Request
Content
Task
Content
Inspection
App C
Task
Encode
App B
Task
Publish
Wor...
Execution Flow
App A
Conductor
App A
Task
Request
Content
Task
Content
Inspection
App C
Task
Encode
App B
Task
Publish
Wor...
Execution Flow
App A
Conductor
App A
Task
Request
Content
Task
Content
Inspection
App C
Task
Encode
App B
Task
Publish
Wor...
Execution Flow
App A
Conductor
App A
Task
Request
Content
Task
Content
Inspection
App C
Task
Encode
App B
Task
Publish
Wor...
Execution Flow
App A
Conductor
App A
Task
Request
Content
Task
Content
Inspection
App C
Task
Encode
App B
Task
Publish
Wor...
Execution Flow
App A
Conductor
App A
Task
Request
Content
Task
Content
Inspection
App C
Task
Encode
App B
Task
Publish
Wor...
Workers
Worker 1
Worker 2
Worker 3
Worker n
...
Management/
Execution Service
Task Queues
Orchestrator
Trigger
Schedule
Ta...
Features
〉 Conditional (If...Then...Else)
〉 Fork / Join
〉 Dynamic Tasks and Forks
〉 Sub Workflow
〉 Wait
〉 Versioning
〉 HTTP/Service...
Runtime
〉 Pause | Resume
〉 Skip Tasks, Restart
〉 Error / Failure Handling
〉 UI
〉 Manage definitions
〉 Search Executions by...
Conductor @ Netflix
〉 In production > 1 year
〉 ~100 Process Flows
〉 ~200 Tasks / Services
〉 Avg. Tasks per workflow: 6
〉 L...
Roadmap - 2017
〉 Eventing
〉 SQS and SNS
〉 Dyno-Queues
〉 Task Execution Log
〉 Unit Testing Framework
〉 Python Client
〉 Feat...
Questions?
Feedback / Issues / Questions
〉 https://github.com/Netflix/conductor
〉 https://github.com/Netflix/dyno-queues
C...
Thank You
Netflix conductor
Upcoming SlideShare
Loading in …5
×

Netflix conductor

8,880 views

Published on

Conductor is a microservices orchestration framework.
https://github.com/Netflix/conductor

Published in: Technology
  • Hello! I can recommend a site that has helped me. It's called ⇒ www.WritePaper.info ⇐ So make sure to check it out!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Legitimate top paying survey, ❤❤❤ https://t.cn/A6ybK3XL
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • To get professional research papers you must go for experts like ⇒ www.HelpWriting.net ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sie können Hilfe bekommen bei ⇒ www.WritersHilfe.com ⇐. Erfolg und Grüße!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating direct: ❤❤❤ http://bit.ly/2F90ZZC ❤❤❤
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Netflix conductor

  1. 1. Netflix Conductor A Microservices Orchestrator Viren Baraiya, Vikram Singh
  2. 2. Content Platform Engineering - CPE 〉 Studio In the Cloud 〉 Content Ingest from Studio Partners 〉 Title Setup - Making it live on Netflix.com 〉 Localization
  3. 3. CPE - Processes (some of many) 〉 Content Ingest & Delivery 〉 Title Setup 〉 IMF* Deliveries 〉 Encodes and Deployments 〉 Content Quality Control 〉 Content Localization * IMF - Interoperable Master Format
  4. 4. Once Upon A Time ... 〉 Peer to Peer Messaging 〉 10’s MM messages per day 〉 Process flows embedded in applications 〉 Lack of control (STOP deployment!) 〉 Lack of visibility into progress
  5. 5. Example
  6. 6. Peer to Peer Application C Application BApplication BApplication A Request Content Content Inspection Result Encode Publish Events / API calls Events / API calls Events / API calls
  7. 7. Peer to Peer Application C Application BApplication BApplication A Request Content Content Inspection Result Encode Publish Events / API calls Events / API calls Events / API calls 〉 Logical flow is not easily trackable 〉 Modifying steps is not easy (tightly coupled) 〉 Controlling flow is not possible 〉 Reusing tasks is not trivial
  8. 8. 〉 Orchestration Engine
  9. 9. 〉 Orchestration Engine 〉 Open Source (Apache 2.0)
  10. 10. Conductor - Design Goals 〉 BYO Task (Reuse existing code) 〉 REST/HTTP support 〉 Extensible and Hackable 〉 JSON based DSL to define blueprint 〉 Scale Out Horizontally 〉 Visibility, Traceability & Control
  11. 11. Same Flow - New Flavor Request Content Content Inspection Result Encode PublishStart Stop Conductor Application A Task Request Content Application B Task Content Inspection Application C Task Encode Application B Task Publish OrchestrationExecution
  12. 12. Architecture API Workflows Metadata Tasks SERVICE Workflow Service Task Service Decider Service Queue Service STORE Storage (Dynomite) Start and manage workflows Define blueprints and tasks Gets tasks from queue and execute Index (Elasticsearch)
  13. 13. Scaling up Conductor 〉 Peer-to-Peer - Scale horizontally 〉 Stateless server - state is persisted in database 〉 Storage scalability : Dynomite 〉 Workload scale: Dyno-Queues
  14. 14. Storage Dynomite 〉 Generic Dynamo implementation (Redis, Memcache) 〉 Multi-datacenter 〉 Highly available 〉 Peer-to-Peer Elasticsearch 〉 Indexing workflow and task executions 〉 Verbose logging of worker executions
  15. 15. Dyno-Queues 〉 Distributed lock free queues used by Conductor 〉 OSS 〉 Apache 2.0 License 〉 https://github.com/Netflix/dyno-queues 〉 Delayed Queues 〉 Loose priorities and FIFO
  16. 16. Task 〉 Reusable stateless components 〉 System Tasks (Fork / Join, Decision, etc.) 〉 Remote Workers (Java, Python, lang. agnostic) Concepts
  17. 17. Concepts Task 〉 Reusable stateless components 〉 System Tasks (Fork / Join, Decision, etc.) 〉 Remote Workers (Java, Python, lang. agnostic) Workflow 〉 Sequence of tasks and 〉 Control structure 〉 Input / Output
  18. 18. Tasks Definition 〉 Retries 〉 Timeouts 〉 Documentation for Input / Output 〉 Registry
  19. 19. Task Definition - Example { "name": "encode_task", "retryCount": 2, "timeoutSeconds": 3600, "inputKeys": [ "fileLocation", "encodeRecipie", "outputLocation" ], "timeoutPolicy": "TIME_OUT_WF", "retryLogic": "FIXED", "retryDelaySeconds": 60, "responseTimeoutSeconds": 3600 }
  20. 20. Workflow Definition 〉 Logical Flow of Tasks 〉 Error / Failure Handling 〉 Input / Output transformation 〉 JSONPath Based 〉 Versioning 〉 Registry
  21. 21. Workflow - Example { "name": "encode_and_deploy", "description": "Encodes a file and deploys to CDN", "version": 1, "tasks": [ { "name": "encode", "taskReferenceName": "encode", "type": "SIMPLE", "inputParameters": { "fileLocation": "${workflow.input.fileLocation}"} }, { "name": "deploy", "taskReferenceName": "d1", "type": "SIMPLE", "inputParameters": { "fileLocation": "${encode.output.encodeLocation}"} } ], "outputParameters": { "cdn_url": "${d1.output.location}" } }
  22. 22. Input / Output 〉 Input to tasks are transformed 〉 Refer Input / Outputs from 〉 Task 〉 Workflow 〉 Complex JSON transformations using JSONPath
  23. 23. Example { "inputParameters": { "movieId": "${workflow.input.movieId}", "url": "${workflow.input.fileLocation}", "lang": "${loc_task.output.languages[0]}", "http_request": { "method": "POST", "url": "http://example.com/${loc_task.output.fileId}/encode", "body": { "recipe": "${workflow.input.recipe}", "params": { "width": 100, "height": 100 } } } } }
  24. 24. Example { "inputParameters": { "movieId": "${workflow.input.movieId}", "url": "${workflow.input.fileLocation}", "lang": "${loc_task.output.languages[0]}", "http_request": { "method": "POST", "url": "http://example.com/${loc_task.output.fileId}/encode", "body": { "recipe": "${workflow.input.recipe}", "params": { "width": 100, "height": 100 } } } } } Get movieId from workflow Input
  25. 25. Example { "inputParameters": { "movieId": "${workflow.input.movieId}", "url": "${workflow.input.fileLocation}", "lang": "${loc_task.output.languages[0]}", "http_request": { "method": "POST", "url": "http://example.com/${loc_task.output.fileId}/encode", "body": { "recipe": "${workflow.input.recipe}", "params": { "width": 100, "height": 100 } } } } } Get movieId from workflow Input JSONPath Expressions
  26. 26. Execution Flow Putting It All Together
  27. 27. Execution Flow App A Conductor App A Task Request Content Task Content Inspection App C Task Encode App B Task Publish Workflow / Task Service Decider/Queue Service 1. Start content_ingest workflow Storage Task Queues App B
  28. 28. Execution Flow App A Conductor App A Task Request Content Task Content Inspection App C Task Encode App B Task Publish Workflow / Task Service Decider/Queue Service 1. Start content_ingest workflow 2. Get Workflow Definition Storage Task Queues App B
  29. 29. Execution Flow App A Conductor App A Task Request Content Task Content Inspection App C Task Encode App B Task Publish Workflow / Task Service Decider/Queue Service 1. Start content_ingest workflow 2. Get Workflow Definition 3. Schedule Task Storage Task Queues App B
  30. 30. Execution Flow App A Conductor App A Task Request Content Task Content Inspection App C Task Encode App B Task Publish Workflow / Task Service Decider/Queue Service 1. Start content_ingest workflow 2. Get Workflow Definition 3. Schedule Task 4. Put in Queue Storage Task Queues App B
  31. 31. Execution Flow App A Conductor App A Task Request Content Task Content Inspection App C Task Encode App B Task Publish Workflow / Task Service Decider/Queue Service 1. Start content_ingest workflow 2. Get Workflow Definition 3. Schedule Task 4. Put in Queue 5. Poll For task Storage Task Queues App B
  32. 32. Execution Flow App A Conductor App A Task Request Content Task Content Inspection App C Task Encode App B Task Publish Workflow / Task Service Decider/Queue Service 1. Start content_ingest workflow 2. Get Workflow Definition 3. Schedule Task 4. Put in Queue 5. Poll For task 6. Execute & update task status Storage Task Queues App B
  33. 33. Workers Worker 1 Worker 2 Worker 3 Worker n ... Management/ Execution Service Task Queues Orchestrator Trigger Schedule Task HTTP Database Index HTTP Update Task Status Queue Poll
  34. 34. Features
  35. 35. 〉 Conditional (If...Then...Else) 〉 Fork / Join 〉 Dynamic Tasks and Forks 〉 Sub Workflow 〉 Wait 〉 Versioning 〉 HTTP/Service Calls 〉 Input / Output Transformation (JSONPath based) Blueprint Features
  36. 36. Runtime 〉 Pause | Resume 〉 Skip Tasks, Restart 〉 Error / Failure Handling 〉 UI 〉 Manage definitions 〉 Search Executions by Payload 〉 Visualize flows 〉 Metrics, Metrics, Metrics…
  37. 37. Conductor @ Netflix 〉 In production > 1 year 〉 ~100 Process Flows 〉 ~200 Tasks / Services 〉 Avg. Tasks per workflow: 6 〉 Largest : 48 Tasks 〉 ~4 MM Executions
  38. 38. Roadmap - 2017 〉 Eventing 〉 SQS and SNS 〉 Dyno-Queues 〉 Task Execution Log 〉 Unit Testing Framework 〉 Python Client 〉 Feature Stacks
  39. 39. Questions? Feedback / Issues / Questions 〉 https://github.com/Netflix/conductor 〉 https://github.com/Netflix/dyno-queues Contacts 〉 Viren Baraiya <vbaraiya@netflix.com> 〉 Vikram Singh <visingh@netflix.com> 〉 Prosenjit Bhattacharyya <pbhattacharyya@netflix.com>
  40. 40. Thank You

×