Presented at QCon NYC 2016. The Netflix API is the front-door for almost all device/UI requests from 1000+ device types to the Netflix backends. It serves everything from movie and show recommendations, profile, sign-up, and A/B test related functionality, to bookmarks and licenses for playback. Because all devices use this API, and because Netflix runs on devices of widely varying sizes and interaction models, it has served us well to enable a platform against which device teams write server-side scripts. Using Netflix as an example, the goal of this talk is to explore situations in which server-side scripting is a good solution for applications. I will describe our first approach, which uses Groovy scripts. I will detail how the scripts are uploaded and can make use of shared modules. This approach allows for high flexibility and performance as well as high developer velocity, at the expense of added risk of injecting scripts into running servers. I will then dive into a new approach that will isolate the scripts into their own containers without compromising the original goals and will allow teams to write scripts in node.js, a language that is more natural for them.

1. The Netflix API Platform for
Server-Side Scripting

2. The risks of modifying running production servers
Problem identified: new
servers aren’t coming up
healthy!
Ugh! There’s a problem.
Errors from API are up.

3. Escalating customer impact
Stream starts per second more and
more off.
Expected value
Actual value

4. Resolving the issue
Finally root-caused!
Now restarting all
unhealthy servers.
Back to
normal!

5. Resolving the issue
Stream starts per
second also back to
normal.
Expected value
Actual value

6. The Netflix API

7. Access to Netflix
mid-tier services

8. Today’s system

9. Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
Network
boundary
API Server JVM
Today’s system (simplified)

10. What we need

11. What we need

12. What we need

13. Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
scripts
scripts
scripts
scripts
...
scripts
scripts
scripts
scripts
Network
boundary
API Server JVM
~700 active
Today’s architecture
groovy

14. Flexibility for devices
[...]
Device1VideoCommon. formatKidsSeason (apiRequest,[...], imageUrl)
[...]
[...]
Device2Common. formatAllSeasons([...])
[...]
[...]
dataPublishingService.getShowFeedbackBuilder(user, video)
[...]

15. What we need

16. Developer Velocity:
Decoupled deployments of versions
n+3
i+4
i+1
i+2i+3
i
n+2
n+1
n
k+1
k j
j+1
l

17. What we need

18. Challenge #1: Resiliency

19. Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
script
script
...
script
script
Network
boundary
API Server JVM
Resiliency in today’s system
Strong
resiliency
with Hystrix
What about
resiliency on
this side?
groovy

20. Example: memory usage
Periodic
cleanup
New upload
increases
memory usage.

22. Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
script
script
...
script
script
Network
boundary
API Server JVM
1-2 years ago
few, small
scripts
fewer uploads
groovy

23. Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
script/app
script/app
script/app
script/app
...
script/app
script/app
script/app
script/app
Network
boundary
API Server JVM
Today
script/app
script/app
~700 more
complex
scripts/apps,
10-50 uploads
per day
groovy

24. Streaming Hours Per Year in Billions

25. Changing risk profile
→
→

26. Lack of process isolation is a growing risk.

27. Some possible mitigations

28. Velocity vs. Resiliency

29. Moving toward our ideal API:
What will change

30. Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
node script
node script
...
node script
node script
Network
boundary API Server JVM
The (near) future
node.js
process
isolation

31. Why containers?

32. Isolated failures: scripts don’t affect each other
(usually)
API
Temporarily
unavailable!

33. Independent autoscaling
API

34. Fast startup

35. Challenge #2: Great developer experience

36. Step-through-
debugging (today)

37. Docker Machine
local
project
Local Container
live reload file watcher
docker build / run
File watcher
agent
Proxy
Network
Agent
Local development (future)
node-inspector
debugger

38. Run-time debugging/optimization (today)
Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
script
script
script
script
...
script
script
script
script
Network
boundary
API Server JVM
script
script
Problems hard to
root cause, hard to
measure/optimize
performance
groovy

39. Script → API interaction (today)
API
device
server-side
script
device client

40. Script → API interaction (future)
API
device
server-side
script

41. Platform for device teams

42. Default configuration

43. Default configuration

44. Default configuration

45. Titus
ATLAS
NeWT: Netflix Workflow Toolkit (CLI)

46. Corresponding UI
Versioning
Easy
access to
instances
Rollback

47. Initial impressions

48. Client A
Client B
Client C
Client E
Netflix
Microservices
node script
Network
boundary API Server JVM
First end-to-end implementation and shadow traffic

49. Client A
Client B
Client C
Client E
Netflix
Microservices
node script
Network
boundary API Server JVM
Problem isolation (ex: memory leak)
Memory leak makes
RSL blow up.
Clearer idea of
where the problem
is.

50. node.js
Client A
Client B
Client C
Client E
Netflix
Microservices
node script
Network
boundary API Server JVM
Problem isolation (ex: memory leak)
Same with node script.

51. Request tracing: clearer picture of fan-out

52. Js
(mostly)
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
node script
node script
...
node script
node script
Network
boundary API Server JVM
node.js
How not to compromise what we’re good at

53. What we need

55. Other Netflix talks at QCon New York

56. Thank you!

58. Script management

59. Outtakes

60. Limited device-server chattiness

61. Operational insights

62. Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
node script
node script
...
node script
node script
Network
boundary API Server JVM
node.js
process
isolation