Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2h3bAvP. Haley Tucker discusses how other systems may affect Netflix' services, strategies to protect their systems and make sure they won't fail even if things go wrong. Filmed at qconsf.com. Haley Tucker works on the Playback Features team at Netflix, responsible for ensuring that customers receive the best possible viewing experience every time they click play. Her services fill a key role in enabling Netflix to stream amazing content to 65M+ members on 1000+ devices.

1. The Forces That Disrupt Netflix
Nov. 7, 2016Haley Tucker

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-resilience

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. ACROBAT
FLEA
our world
parallel world

5. # A distributed system is
one in which the failure
of a computer you didn't
even know existed can
render your own computer
unusable.
--Leslie Lamport

6. ACROBAT
FLEA
our world
parallel world
computing
ENGINEER

8. PROLOGUE
DISTRIBUTED SYSTEMS

10. Proxy/Routing
DECOMPOSING THE MONOLITH
Devices
Netflix
Service
Netflix
Service
Edge
Service
Traffic
Netflix
Playback
Service
Netflix
Playback
Service
Edge
Service
Edge
Service
Edge
Service
Playback
Service
Traffic

11. Notes on Distributed Systems for Young Bloods
# Distributed
systems are
different because
they fail often.
--Jeff Hodges

12. TABLE OF CONTENTS
CHAPTER 1: THE WEIRD DATA IN THE CATALOG
• Metadata impacts on availability
CHAPTER 2: THE VANISHING OF CRITICAL SERVICES
• Crashing services and cascading failures
CHAPTER 3: THE THROTTLE
• Latency spikes and the impact of fallbacks
FORCES AT WORK

14. Whoops, something went wrong…
Netflix Streaming Error
We’re having trouble playing this title right now. Please try again later or select a different title.

15. CHAPTER ONE
THE WEIRD DATA IN THE CATALOG

17. 45 MINUTES!!
Clock, by heyyobecky4lyfe, Tumblr

19. VIDEO METADATA
ARCHITECTURE
Video
Metadata
Service
Amazon S3
Source
System
Source
System
Netflix
Services
Netflix
Services
Netflix
Services
Netflix
ServicesNetflix
Service
Traffic

20. Amazon S3
Netflix
Playback
Service
{
String msg = “This should never
happen!”;
throw new IllegalStateException(msg);
}

21. MITIGATION
BLAST RADIUS
Explosion, CC BY 2.0, Andrew Kuznetsov 2008, Flikr

22. Amazon WS Global Infrastructure

23. Amazon WS Global Infrastructure
STAGGERED ROLLOUT

24. Pager
Diagnosis?

26. Canary, CC BY 2.0, Steve P2008 2014, Flikr
PREVENTION
CANARIES

27. TRADITIONAL CANARY
Canary
(New Code)
Baseline
(Old Code)
TrafficTraffic
Video
Metadata
Service
Amazon S3
Netflix
Services
Netflix
Services
Netflix
Services
Netflix
ServicesNetflix
Service
Source
System
Source
System
Traffic

28. CONSISTENCY
VALID STATE TRANSITIONS

29. DATA CANARY
Netflix
Services
Netflix
Services
Netflix
Services
Netflix
Services
Video
Metadata
Service
Amazon S3
Source
System
Source
System
Netflix
Service
Netflix
Data
Canary
Service
Data
Tester
Netflix
Service
Traffic

30. Australia with AAT, CC BY-SA 2.0, Ssolbergj 2010, Wikimedia
SEEING RETURNS

31. Verify consistency prior to
applying state changes.
…one tool is a data canary.

32. CHAPTER TWO
THE VANISHING OF CRITICAL SERVICES

35. # A distributed system is
one in which the failure
of a computer you didn't
even know existed can
render your own computer
unusable.
--Leslie Lamport

36. Proxy/Routing
Devices
LOG DATA
Log Data Service
Traffic
Cassandra
Playback ServiceNetflix
Playback
Service
Netflix
Playback
Service
Edge
Service
Edge
Service
Edge
Service
Playback
Service
Traffic

38. Proxy/Routing
Devices
Proxy
Log Data Service
Traffic
Cassandra
Playback ServiceNetflix
Playback
Service
Netflix
Playback
Service
Edge
Service
Edge
Service
Edge
Service
Playback
Service
Traffic

40. Proxy/Routing
Devices
CASCADING FAILURE
Log Data Service
Traffic
Cassandra
Playback ServiceNetflix
Playback
Service
Netflix
Playback
Service
Edge
Service
Edge
Service
Edge
Service
Playback
Service
Traffic

41. {
throw new OutOfMemoryError();
}
Log Data Service
Cassandra
Playback Service

42. PREVENTION
MANAGING RESOURCE CONSTRAINTS
Whatever you ask, CC BY-SA 2.0, Kreg Steppe 2008, Flikr

43. Astronomical Clock, CC BY 2.0, Andrew Fleming 2011, Flikr
REDUCE SURFACE AREA

44. 1
Keep Only
Dependencies
which are
Necessary
SO MANY JARS!!

45. 2 LIMIT “MAGIC”
Magic, CC BY-ND 2.0, Daniel Lee 2013, Flikr

46. 3
Medusa Kill Switch, CC BY-NC-ND 2.0, Scott Hart 2013, Flikr
ADD KILL SWITCHES

47. DEV
FAVOR IMMUTABILITY
Playback Service
TEST
Playback Service
PROD
Playback Service
4

48. try {
remoteService.call();
} catch( Throwable t ){
//Oops!
System.exit(1);
}
Log Data
Service
Cassandra
Playback
Service
Proxy/Routing
Traffic

49. It's Electric, CC BY ND 2.0, Alan Hochberg 2008, Flikr
MITIGATION
CIRCUIT BREAKERS

51. Wrecking Ball in Building, CC BY 2.0, Jason Eppink 2008, Flikr
FAILURE TESTING

52. Proxy/Routing
Devices
FAILURE TESTING
Log Data Service
Traffic
Cassandra
Playback Service Automating Chaos
Experiments in
Production
by Ali Basiri
Applying Failure
Testing Research
@Netflix
by Kolton Andrus and
Peter Alvaro

53. Manage resource constraints by
reducing surface area.
Leverage circuit breakers and
rigorously test failures.

54. CHAPTER THREE
THE THROTTLE

56. Proxy/Routing
Devices
PLAYBACK ARCHITECTURE
Edge Service Edge Service Edge Service
Playback Service
Traffic Traffic
URL Service

57. NETFLIX CLIENT JARS
Playback Service
URL
Service
URL Client
Circuit-breakers and Fallbacks
Metrics Retries and Timeouts
RPCService Discovery

59. Playback
Service
Traffic Concurrent
Requests
Throttled Requests
(HTTP 503)
THROTTLING

61. }
System.gc();
}
URL
Service
Playback Service
Edge Service
Proxy/Routing
Traffic

62. NETFLIX CLIENT JARS
Playback Service
URL
Service
URL Client
Circuit-breakers
Metrics
Retries and
Timeouts
RPCService Discovery
Heavy
Fallback

63. FALLBACK TESTING
With 100% Fallback,
CPU held at 90%
15 RPSNo fallback, CPU held at 90%
58 RPS
Siege: https://github.com/JoeDog/siege

64. SELECTING FALLBACKS
CACHESTATIC
FALLBACK
SERVICE

65. URL
Service
Playback Service
Edge Service
Proxy/Routing
Traffic
}
return Response
.status(503)
.build();
}

66. REQUEST BUCKETING
NON-CRITICAL
Experience or
Performance
Impact
CRITICAL
Customer
Streaming
Impact
Fire Buckets at Oakworth Statione, CC BY 2.0, Tim Greene 2015, Flikr

67. APPLICATION SHARDING
Non-Critical
Playback Service
Proxy/Routing
Devices
Edge Service Edge Service Edge Service
Traffic Traffic
URL Service
Critical Playback
Service
Non-Critical URL
Service

68. CRITICAL
Country Road at Sunrisee, CC BY-SA 2.0, Susanne Nilssone 2014, Flikr

69. NON-CRITICAL
Traffice, CC BY-NC 2.0, jonbgeme 2008, Flikr

70. APPLICATION SHARDING
Non-Critical
Playback Service
Proxy/Routing
Devices
Edge Service Edge Service Edge Service
Traffic Traffic
Critical Playback
Service
URL Service
Non-Critical URL
Service

71. No heavy fallbacks!!
Fallbacks should be light and fast.
Shard your application based on
operational characteristics.

72. EPILOGUE
KEY TAKEAWAYS

73. KEY TAKEAWAYS
CHAPTER 1: THE WEIRD DATA IN THE CATALOG
• Verify consistency prior to applying state changes.
• One tool is a data canary.
CHAPTER 2: THE VANISHING OF CRITICAL SERVICES
• Manage resource constraints by reducing surface area.
• Leverage circuit breakers and rigorously test failures.
CHAPTER 3: THE THROTTLE
• No heavy fallbacks!! Fallbacks should be light and fast.
• Shard your application based on operational characteristics.

74. The unexpected will happen.
Plan to fail.

75. PARTING THOUGHT
DISTRIBUTED SYSTEMS
SOCIAL

76. Questions?
Haley Tucker

77. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-resilience