RedisConf17 breakout session

1. Making Non-Distributed Databases, Distributed
- Ioannis Papapanagiotou, PhD
- Gim Mahasintunan
- Viren Baraiya

3. Dynomite/Redis Ecosystem
● Dynomite - Proxy Layer
● Dyno - Redis based Client
● Dynomite-manager - Ecosystem Orchestrator
● Dynomite-Explorer - Redis based UI
● Dyno-queues - Redis based keys
● Conductor - Workflow engine
● NDBench - Netflix Data Benchmark (for Redis)

4. ● Needed a data store:
o Scalable & highly available
o High throughput, low latency
o Netflix use case is active-active
● Master-slave storage engines:
o Do not support bi-directional replication
o Cannot withstand a Monkey attack
o Cannot easily perform maintenance
Problems & Observations

5. What is Dynomite?
● A framework that makes non-distributed data
stores, distributed.
Features: highly available, automatic failover, node
warmup, tunable consistency, backups/restores

6. Dynomite @ Netflix
● Running around 2.5 years in PROD
● 70 clusters (100% Y/Y)
● ~1000 nodes used by internal microservices
● Microservices based on Java, Python,
NodeJS

7. Pluggable Storage Engines
● Layer on top of a non-distributed key value data store
○ Peer-peer, Shared Nothing
○ Auto Sharding
○ Multi-datacenter
○ Linear scale
○ Replication(Encrypted)
○ Gossiping

8. Replication
● A client can connect to any node on
the Dynomite cluster when sending
requests.
o If node owns the data,
▪ data are written in local
data-store and
asynchronously replicated.
o If node does not own the data
▪ node acts as a coordinator
and sends the data in the
same rack & replicates to
other nodes in other racks
and DC.

9. ● Each rack contains one
copy of data, partitioned
across multiple nodes in
that rack
● Multiple Racks == Higher
Availability (HA)
Topology

10. Dynomite on the Cloud
Discovery Service
Insights (Metrics)
Continuous Delivery
Healthcheck
Backups & Restores
Dynomite Manager
RESP = Redis Serialization Protocol
REST/HTTP

11. Dyno Client - Java API
● Connection Pooling
● Load Balancing
● Effective failover
● Pipelining
● Scatter/Gather
● Metrics, e.g. Netflix Insights

12. Dyno Load Balancing
● Dyno client employs token
aware load balancing.
● Dyno client is aware of the
cluster topology of Dynomite
within the region,
can write to specific node
using consistent
hashing.

13. Moving across Storage Engines

14. Netflix Data Benchmark
● Benchmarking Dynomite with
Redis on the Cloud

15. Netflix Data Benchmark for Redis
● Dynamically change the benchmark configurations,
○ perform tests along with our production microservices.
● Be able to integrate with platform cloud services
○ dynamic configurations, discovery, metrics, etc.
● Run for an infinite duration in order to introduce failure scenarios
● Provide pluggable patterns and loads.
● Support different client APIs.
● Deploy, manage and monitor multiple instances from a single entry point.

17. How everything comes together?

18. Netflix Data Explorer - Dynomite
● Exploring Netflix Data Sources
● Providing a UI for Dynomite and Redis

19. Netflix Data Explorer - Use Cases
Netflix needed a client to satisfy the following requirements:
● Support Redis API
● Avoid blocking calls (e.g. Redis KEYS *)
● UI needs to scale to millions of keys
● Customizable UI
● Ability to share UI components amongst Netflix projects
● Provide extensive logging for audit trail purposes

20. Architecture

21. Cluster Listing
● Multi-tenant system
● Each application
has a dedicated
Dynomite cluster
● Access controls
restrict cluster
visibility

22. Scanning for Keys

23. JSON and XML

24. Creating Complex Types

25. TTL Support

26. Audit Trail and Logging
eu-west
c
b
a
us-east
c
b
a
us-west
b
a
Elasticsearch

27. Conductor
● Orchestration Engine
● Redis for Storage and Queues
● Open Source (Apache 2.0)

28. Orchestration - Use Cases
● Content Ingest & Delivery
● Title Setup
● Studio Deliveries
● Content Quality Checks
● Content Localization

29. Once Upon A Time...
● Peer to Peer Messaging
● 10’s MM messages per day
● Process flows embedded in applications
● Lack of control (STOP deployment!)
● Lack of visibility into progress

30. Peer to Peer
Application C Application BApplication BApplication A
Request Content Content Inspection Result Encode Publish
Events / API calls Events / API calls Events / API calls

31. Peer to Peer
Application C Application BApplication BApplication A
Request Content Content Inspection Result Encode Publish
Events / API calls Events / API calls Events / API calls
● Logical flow is not easily trackable
● Modifying steps is not easy (tightly coupled)
● Controlling flow is not possible
● Reusing tasks is not trivial

32. Conductor
● BYO Task (Reuse existing code)
● REST/HTTP support
● Extensible and Hackable
● JSON based DSL to define blueprint
● Scale Out Horizontally
● Visibility, Traceability & Control
● UI to monitor and manage workflows (node.js/react)

33. Same Flow - New Flavor
Request
Content
Content
Inspection
Result Encode PublishStart
Stop
Conductor
Application A
Task
Request
Content
Application B
Task
Content
Inspection
Application C
Task
Encode
Application B
Task
Publish
OrchestrationExecution

34. Developer Workflow

35. High Level Architecture
API
Workflows Metadata Tasks
SERVICE
Workflow Service Task Service
Decider Service Queue Service
STORE
Storage (Redis /Dynomite)
Start and manage
workflows
Define blueprints
and tasks
Gets tasks from
queue and execute
Index (Elasticsearch)

36. Conductor - Scale
● Peer-to-Peer - Scale horizontally
● Stateless server - state is persisted in Redis
● Storage scalability : Dynomite
● Workload scale: Dyno-Queues

37. Storage Layer
● Dynomite
○ Generic Dynamo implementation (Redis, Memcache)
○ Multi-datacenter
○ Highly available
○ Peer-to-Peer
● Elasticsearch
○ Indexing workflow and task executions
○ Verbose logging of worker executions

38. Dyno-Queues
● Distributed lock free queues used by Conductor
● OSS
○ Apache 2.0 License
○ https://github.com/Netflix/dyno-queues
● Delayed Queues
● Loose priorities and FIFO
● Redis based
● At-least once delivery

39. Dyno-Queues

40. Conductor @ Netflix
● In production > 1.5 year
● Used by Content Platform Engineering
○ Content Ingest & Encoding
○ Content Processing
● ~150 Process Flows & ~300 Tasks / Services
● 1+ MM Executions / Month

41. More information
● Dynomite Ecosystem:
o https://github.com/Netflix/dynomite
o https://github.com/Netflix/dyno
o https://github.com/Netflix/dyno-queues
o https://github.com/Netflix/dynomite-manager
● NDBench:
o https://github.com/Netflix/ndbench
● Conductor:
o https://github.com/Netflix/Conductor
● Chat:
o https://gitter.im/Netflix/dynomite
o https://gitter.im/Netflix/conductor