Traditionally, a tug of war has existed between service reliability (availability, consistency, and fault tolerance), and engineering velocity. Increasing speed to fuel product innovation has meant making reliability tradeoffs. In this talk Yunong will describe how abstracting away the infrastructure and common platform components by using a Platform as a Service (Paas) model can enable service owners to increase both velocity and reliability.

1. The Paved PaaS to
Microservices
Yunong Xiao,
Principal Software Engineer, Netflix
yunong@netflix.com, @yunongx, http://yunong.io
November, 10, 2017

2. 110 million customers
in over 190 countries
streaming 125 million hrs/day

3. Subscriber Growth
20M
38M
56M
74M
92M
110M
2011 2012 2013 2014 2015 2016 2017

9. Microservices

12. Discovery
Sign-up
Playback
…
Netflix Edge API
http://api.netflix.com

14. 1000s/year
A/B

15. Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!
Push!

17. 1000s of Routes
Pushing multiple
times/week
TV
iOS
Android
Windows
Browsers
Discovery
Playback
Non-
member
…
Backend
Service A
Backend
Service B
Backend
Service C
Backend
Service N
…
Clients Standalone Services Edge API Backend Services

18. Goals
Velocity Reliability

19. –Wikipedia
“Platform as a service (PaaS)… allows customers to
develop, run, and manage applications without the
complexity of
building and maintaining the infrastructure and
platform…”
What is a Platform as a Service (Paas), Anyway?

20. Know Your Customer

21. Our PaaS to Achieving Both
1. Standardized components
2. Preassembled platform
3. Automation and tooling

22. 1. Standardized Components

23. What’s Inside a Microservice?
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing

24. Why Standards?

25. Mélange of RPC

26. Microservice Interactions
µServiceClient
uService
µService
µService
µService
…
µService
…
µService

27. Failure: When not If

28. – Managers everywhere
“Is it fixed yet?”
Mean Time to Detection (MTTD)
Mean Time to Repair (MTTR)

29. N Flavors of RPC
µServiceClient
uService
µService
µService
µService
…
µService
…
µService

30. One Standard RPC
µServiceClient
uService
µService
µService
µService
…
µService
…
µService

31. Benefits of Standardizing
Consistency Leverage Interoperability Quality Support
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing

32. But I'm a Snowflake...
Freedom Responsibility&
Off-road Innovation Reintegrate BurdenVacuumNew

33. 2. Preassembled Platform

34. Assembly Required
RPC
Discovery
Registration
Runtime
OS
Configuration
Metrics
Logging
Tracing
Dashboards
Alerts
Stream
Processing

37. Getting Out of the Blocks
Docs
Copy/paste
Which versions?
Initialization
Configuration
Missing components
Days or weeks

38. Velocity Reliability
Product Innovation
Not a single line of business logic!

39. Preassembled Platform
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing

40. Out of the Box

41. Component Management

43. Insights

44. Application, system, and runtime metrics & logs

45. Consistent application, system, and runtime
metrics & logs
Reduces MTTD & MTTR

46. Configures and Initializes Correctly

47. Configures and Initializes Correctly
BOEING 747-400 NORMAL PROCEDURES CHECKLIST
POWER UP / SAFETY CHECK
First Officer Captain
CIRCUITBREAKERS...………………...CHECKED
BATTERY……………...………………….………ON
STANDBY POWER………………...………..AUTO
HYDRAULIC DEMAND PUMPS….………..…..OFF
WINDSHIELD WIPERS....……………….……..OFF
ALTERNATE FLAPS AND GEAR…………….OFF
GEAR LEVER…….………….....…………..DOWN
FLAPS…..……………………………....CHECKED
APU….………………………………...….RUNNING
ELECTRICAL SYSTEM....……SET/APU AVAIL ON
APU BLEED AIR…...……………………………ON
ISOLATION VALVES…………………………OPEN
PACKS………………...……………………NORMAL
PREFLIGHT
First Officer Captain
EMERGENCY EQUIPMENT…….…….CHECKED
FIRE PROTECTION…………...………CHECKED
INTERRUPT SWITCHES………………………ON
PASSENGER OXYGEN……………..……NORMAL
STAB TRIM CUTOUT SWITCHES………….AUTO
TRANSPONDER…………………………………SET
SOURCE SELECTORS…………………...…….SET
CLOCKS…………………………………………..SET
CRT SELECTORS…………………………NORMAL
PFD………………………………………CHECKED
ND…………………………………………CHECKED
AUTOBRAKES…………………………………RTO
EIU SELECTOR………………………………AUTO
HDG REFERENCE SWITCH….…………NORMAL
FMC MASTER SELECTOR…………………LEFT
GROUND PROX SYSTEM……………CHECKED
BEFORE STARTING
First Officer Captain
HYDRAULIC DEMAND PUMPS…………………….
.............................................AUTO, AUX (1 AND 4)
BRAKE PRESSURE……..……………….NORMAL
FUEL QUANTITY………………..…………. ____KG
FUEL SYSTEM………………………………….SET
X-FEEDS…………………OPEN (1 & 4 CLOSED)
SEAT BELTS SIGN………………………………ON
NOTOC………………………………….CHECKED
SHIPS PAPERS…………………………ON BOARD
PERFORMANCE DATA......…CHECKED AND SET

48. Versions Updates Compatibility

49. Important Questions

50. What’s in and out?

51. Maintenance vs Convenience
RPC Discovery Registration
Runtime OS Configuration
Metrics Logging Tracing
Dashboards Alerts
Stream
Processing
Base Platform

52. Solution: Layers & Flavors
Base platform
Data access
Rendering
Backend
…

53. How to Ensure Platform
Correctness?

54. Test, Test, Test
Unit
Integration
Functional
Cloud
Every PR

56. Component Correctness
Lock down
versions
Test
components
Updates require
PRs
Gate Keeper
RPC Discovery Registration
Runtime OS Configuration
Metrics Logging Tracing
Dashboards Alerts
Stream
ProcessingCustomers

57. How Locked Down is it?

58. Tradeoffs
vs
Flexibility
Reliability
Consistency
Support

59. Stay on paved path!

60. Season to Taste
Config overrides
Startup & shutdown hooks
Access to 3rd party libs
Swap, disable, or configure
components
Raw component access

61. Platform Versioning?

62. API Semantic Versioning
1.2.3
^1.0.0
~1.3.0

63. Use Conventional Changelog

64. 3. Automation and Tooling

65. Ship a Feature

66. Development
Testing
Deployment Operations
Steps

67. Development

70. Env bootstrap Integrate tooling & services Run local & cloud
CLI for common dev experience

71. Local Development
Live reload
Attach debugger
localhost
PROD

72. Testing

73. Testing

74. Testing
Preassembled Platform
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing

75. Preassembled Platform
Pre-Prod
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing

76. Provide First Class Mocks
RPC Discovery Registration Runtime OS Configuration
Metrics Logging Tracing Dashboards Alerts
Stream
Processing
Preassembled Platform

77. Platform Testing API
• Just like a runtime API, need a testing API
• Provide mocks interface for components
• Gets platform out of the loop for providing mocks

79. Deployment

80. “Production is war!”

81. Experience Differences

82. Deploy and Manage Services
• Pre-configured pipelines for deployment and rollback
• Single command deploy to any stack
• Integration for automated canary analysis
• Pre-configured autoscaling

83. Operations

84. Consolidated View

85. RPS
Latency
Error rates
CPU
Memory
Generated Dashboards & Alerts

86. CPU profiling Core dump analysis
Automated Analytics & Tooling

87. Our PaaS to Velocity & Reliability
1. Standardized components
2. Preassembled platform
3. Automation and tooling

88. The Paved PaaS to
Microservices
Yunong Xiao,
Principal Software Engineer, Netflix
yunong@netflix.com, @yunongx, http://yunong.io
November, 10, 2017