At Netflix, we provide a Java-based API that supports the content discovery, sign-up, and playback experience on thousands of device types that millions use around the world every day. As our user base and traffic have grown by leaps and bounds, we are continuously evolving this API to enable the best user experience. In this talk, I will give an overview of how and why the Netflix API has evolved to where it is today and where we plan to take it in the future. I will discuss how we make our system resilient against failures using tools such as Hystrix and FIT, while keeping it flexible and nimble enough to support continuous A/B testing.

1. The Netflix API for a
global service
Katharina Probst
Engineering Manager, API
DevNexus, February 2016

2. What is Netflix?
Stream TV
shows and
movies
anywhere,
any time.

3. Global!
(except China and where we can’t operate for legal reasons)

4. Netflix
Originals

5. Scale
❏ Peak
downstream
traffic in the
US is 37%,
upstream
almost 7%.
❏ 75 Million subscribers worldwide and growing
Source: http://www.sandvine.com/news/global_broadband_trends.asp

6. Netflix API
❏ Architecture
❏ Resiliency
❏ Developer velocity
❏ Tooling and DevOps
❏ Current and future directions
API

7. Netflix API
❏ Architecture
❏ Resiliency
❏ Developer velocity
❏ Tooling and DevOps
❏ Current and future directions
API

8. API
Personali-
zation
Engine
User
Info
Ratings
Similar
Movies
A/B Test
Engine
….
ELB
Zuul
(gateway)

9. RxJava Hystrix
JavaService
Layer
Mid-tier
Services
UI
Teams
Client Server
Internet
Application
/tv/home
API
Team
Service
Teams

10. What is the API used
for?
Examples:
❏ Discovery
❏ Recommendations
❏ Move metadata
❏ Ratings
❏ Sign-up and Profiles
❏ Playback
❏ Bookmarks
❏ DRM
❏ A/B testing
API

11. Direct dependencies on other services

12. Netflix API
❏ Architecture
❏ Resiliency
❏ Developer velocity
❏ Tooling and DevOps
❏ Current and future directions
API

13. Hystrix Primer
❏ Protection from and control over
latency and failure from dependencies
❏ Stop cascading failures in a complex
distributed system
❏ Fall back and gracefully degrade
❏ Fail fast and rapidly recover
https://github.com/Netflix/Hystrix

14. API
Personali-
zation
Engine
User
Info
Ratings
Similar
Movies
A/B Test
Engine
….

15. API
Personali-
zation
Engine
User
Info
Ratings
Similar
Movies
A/B Test
Engine
….
Don’t let this
happen.

16. API
Personali-
zation
Engine
User
Info
Ratings
Similar
Movies
A/B Test
Engine
….
Don’t let this
happen.

17. Fallback
Response
Do this
instead.
API
Personali-
zation
Engine
User
Info
Ratings
Similar
Movies
A/B Test
Engine
….

18. Failure
Injection
Testing
(FIT)
Goal: Study how the
system behaves when
a failures occur (e.g.,
backend service
unreachable).

19. More automated failure testing
Goal: Find groups of service calls that are needed for
success.
http://techblog.netflix.com/2016/01/automated-failure-testing.html

20. Autoscaling & Capacity Management
http://nflx.it/1LvqLUi

21. Autoscaling & Capacity Management
❏ Red: traffic for current week (x-axis)
❏ Black: traffic for previous week for comparison
❏ What happened on February 7? Superbowl!

22. AWS Controls Reactive, does not
scale up fast enough

23. Fine-grained Control with Scryer
Complements AWS Controls
❏ Faster scale-up, improved cost
❏ Use reactive policy for organic scale down

24. Netflix API
❏ Architecture
❏ Resiliency
❏ Developer velocity
❏ Tooling and DevOps
❏ Current and future directions
API

25. Lots of devices, lots of variety

26. Different interaction models

27. And just to make things a little more
interesting….
❏ A/B tests
❏ profiles
❏ localization

28. Add server-side scripting capability
❏ Reduce network chattiness
❏ Support device optimizations
❏ Enable faster development for internal users

29. Discrete HTTP requests pay network tax repeatedly

30. Single, optimized request; pay network tax once
Client data
assembly logic
pushed to server

31. Local MethodRemote API
GET
/users/{user_id}/lists
getLists(userId)

32. ❏ UI (script) changes can happen
independently
❏ Script changes can be pushed to running
servers, so decoupled from API push
schedule
❏ Decoupling leads to greater developer
velocity
Impact on velocity and collaboration

33. Netflix API
❏ Architecture
❏ Resiliency
❏ Developer velocity
❏ Tooling and DevOps
❏ Current and future directions
API

34. Run 1% of your traffic on the new
code and see how it does

35. ❏ Errors: 2xx, 4xx, 5xx
❏ latency
❏ network
❏ busy threads
❏ load, memory consumption
❏ ...
So you’ve run a canary. Now what?
Control Canary

37. Successful canary
red/black push

38. Continuous Delivery with Spinnaker
http://techblog.netflix.com/2015/09/moving-from-asgard-to-spinnaker.html

39. Quickly see status of all clusters
http://techblog.netflix.com/2015/09/moving-from-asgard-to-spinnaker.html

40. Prod is a little different….

41. The things you can do
… with server groups … with instances

42. Script Management

44. Operations

45. Operations

46. Operations

47. Real-time analysis
http://www.slideshare.net/g9yuayon/qcon-talk-on-netflix-mantis-a-stream-processing-system
Submit a query, see requests in real time.

48. Netflix API
❏ Architecture
❏ Resiliency
❏ Developer velocity
❏ Tooling and DevOps
❏ Current and future
directions
API

49. ● > 900 active
endpoints
● ~60 direct
dependencies
● 78 thread pools
● 1000+ threads
● high memory usage
What we’ve
grown to

50. Script isolation & node
❏ Groovy scripts run as
part of API process
❏ UI teams would like to
use other languages
(in particular node.js)
var response = model.get("todos[0..2]
['name','done']");
API remote
service layer
Client libs
UI/device
scripts (node)
Falcor
Services

51. Thin client libraries
❏ Fat client libraries
❏ business logic and
have
❏ multiple dependencies
❏ Move business logic and
dependencies to services
API remote
service layer
Thin client libs
UI/device
scripts (node)
Falcor
Services

52. Remove metadata from API servers
❏ Metadata takes up
significant memory
in API servers
❏ Challenge: reduce
chattiness to
metadata
Metadata
Service
API remote
service layer
Thin client libs
UI/device
scripts (node)
Falcor
Services

53. In the beginning...