History and Future of the Netflix API - Mashery Evolution of Distribution
•
11 likes•3,764 views
Presentation on the history and future of the Netflix API. This presentation walks through how the API was formed, why it needs a redesign and some of the principles that will be applied in the redesign effort. This presentation was given at the Mashery Evolution of Distribution session in San Francisco on June 2, 2011.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (13)
Similar to History and Future of the Netflix API - Mashery Evolution of Distribution
Similar to History and Future of the Netflix API - Mashery Evolution of Distribution (20)
More from Daniel Jacobson
Recently uploaded
Recently uploaded (20)
History and Future of the Netflix API - Mashery Evolution of Distribution
- 1. The Netflix API How Netflix Launched an API and Evolved it to Serve Millions on Hundreds of Devices By Daniel Jacobson
- 2. Who Am I? Director of Engineering for Netflix API since October 2010 Tenure at NPR from 1999 to 2010 Built the custom CMS in 2002 Extended system for RSS and Podcasting Launched the NPR API in 2008 Launched NPR redesign in 2009
- 3. Netflix Overview Netflix offers subscriptions to unlimited streaming movies and TV shows for a very low price About 700 operational employees, 300 engineers More than 23 million subscribers in US and Canada Market capitalization is about $12B Responsible for about 30% of US bandwidth during peak hours (by some accounts)
- 4. How Does the API Fit in at Netflix?
- 5. Original Charter for the Netflix API Expose Netflix metadata and services to the public developer community to “let 1,000 flowers bloom”. That community will build rich and exciting new tools and services to improve the value of Netflix to our customers.
- 6. Netflix API There are currently over 18,000 flowers
- 10. Eat Your Own Dog Food
- 11. New Target Audiences Internal Engineering Teams Netflix product owners Netflix developers Partner Relationships External device manufacturers Public Developer Community 1,000 flowers
- 12. Netflix API
- 13. Some of the many Netflix-ready devices
- 15. Growth of Netflix API
- 16. Netflix API Requests by Audience
- 17. New Charter for the Netflix API Build and maintain an infinitely scalable data distribution pipeline for getting metadata and services from internal Netflix systems to streaming client apps on all platforms in the format and/or delivery method that is most optimal for each app and platform.
- 18. API Personalization Engine User Info Movie Metadata Movie Ratings Similar Movies Reviews A/B Test Engine
- 19. What is the future of the API?
- 20. Growth of Netflix API
- 21. Growth of the Netflix API About 30 Billion requests per month
- 24. <catalog_titles> <number_of_results>1140</number_of_results> <start_index>0</start_index> <results_per_page>10</results_per_page> <catalog_title> <id>http://api.netflix.com/catalog/titles/movies/60021896</id><title short="Star" regular="Star"></title> <box_art small="http://alien2.netflix.com/us/boxshots/tiny/60021896.jpg" medium="http://alien2.netflix.com/us/boxshots/small/60021896.jpg" large="http://alien2.netflix.com/us/boxshots/large/60021896.jpg"></box_art> <link href="http://api.netflix.com/catalog/titles/movies/60021896/synopsis" rel="http://schemas.netflix.com/catalog/titles/synopsis" title="synopsis"></link> <release_year>2001</release_year> <category scheme="http://api.netflix.com/catalog/titles/mpaa_ratings" label="NR"></category> <category scheme="http://api.netflix.com/categories/genres" label="Foreign"></category> <link href="http://api.netflix.com/catalog/titles/movies/60021896/cast" rel="http://schemas.netflix.com/catalog/people.cast" title="cast"></link> <link href="http://api.netflix.com/catalog/titles/movies/60021896/screen_formats" rel="http://schemas.netflix.com/catalog/titles/screen_formats" title="screen formats"></link <link href="http://api.netflix.com/catalog/titles/movies/60021896/languages_and_audio" rel="http://schemas.netflix.com/catalog/titles/languages_and_audio" title="languages and audio"></link> <average_rating>1.9</average_rating> <link href="http://api.netflix.com/catalog/titles/movies/60021896/similars" rel="http://schemas.netflix.com/catalog/titles.similars" title="similars"></link> <link href="http://www.netflix.com/Movie/Star/60021896" rel="alternate" title="webpage"></link> </catalog_title> <catalog_title> <id>http://api.netflix.com/catalog/titles/movies/17985448</id><title short="Lone Star" regular="Lone Star"></title> <box_art small="http://alien2.netflix.com/us/boxshots/tiny/17985448.jpg" medium="http://alien2.netflix.com/us/boxshots/small/17985448.jpg" large=""></box_art> <link href="http://api.netflix.com/catalog/titles/movies/17985448/synopsis" rel="http://schemas.netflix.com/catalog/titles/synopsis" title="synopsis"></link> <release_year>1996</release_year> <category scheme="http://api.netflix.com/catalog/titles/mpaa_ratings" label="R"></category> <category scheme="http://api.netflix.com/categories/genres" label="Drama"></category> <link href="http://api.netflix.com/catalog/titles/movies/17985448/awards" rel="http://schemas.netflix.com/catalog/titles/awards" title="awards"></link> <link href="http://api.netflix.com/catalog/titles/movies/17985448/format_availability" rel="http://schemas.netflix.com/catalog/titles/format_availability" title="formats"></link> <link href="http://api.netflix.com/catalog/titles/movies/17985448/screen_formats" rel="http://schemas.netflix.com/catalog/titles/screen_formats" title="screen formats"></link> <link href="http://api.netflix.com/catalog/titles/movies/17985448/languages_and_audio" rel="http://schemas.netflix.com/catalog/titles/languages_and_audio" title="languages and audio"></link> <average_rating>3.7</average_rating> <link href="http://api.netflix.com/catalog/titles/movies/17985448/previews" rel="http://schemas.netflix.com/catalog/titles/previews" title="previews"></link> <link href="http://api.netflix.com/catalog/titles/movies/17985448/similars" rel="http://schemas.netflix.com/catalog/titles.similars" title="similars"></link> <link href="http://www.netflix.com/Movie/Lone_Star/17985448" rel="alternate" title="webpage"></link> </catalog_title> </catalog_titles>
- 25. Improve Efficiency of API Requests Could it have been 5 billion requests per month? (Assuming everything else remained the same)
- 26. Design for our Key Audiences Some of the many Netflix-ready devices
- 27. Future architecture needs to support key audience first with a trickle down of features to the other audiences Netflix API
- 28. Design for The Audience We Want If our current audience is this size We will designour system for an audience that is this size
- 29. AWS Cloud
- 30. Key Lessons from Netflix Understand target audiences Think big Think mobile Think international Design API for critical audiences first Internalize the API as part of your engineering DNA If building a public API, help them bloom
- 31. The Netflix API Team is Hiring!
Editor's Notes
- In the beginning, the Netflix API only supported the 1,000 flowers. Now there are over 18,000 of them.
- Here are some examples of companies or individual developers that have built apps using the Netflix API.
- Along the way, Netflix launched another developer community-driven program – The Netflix Prize. This was a program that offered $1,000,000 to the first team that could improve the recommendation algorithm by 10%. It took 2.5 years for the prize to be awarded.
- Then streaming started taking off for Netflix and the API became a viable option for getting Netflix streaming onto devices.
- This is what is known as the “eat your own dogfood” model.If you have to eat the dog food, you are more likely to make it taste good. For us, if we build an API for ourselves, others can be assured that it will taste good because we will be eating it too.
- The shift due to streaming results in a redefinition of target audiences for the API.
- So, the API interaction model started to look different with the addition of all of these devices.
- Today, we have hundreds of devices being run off the API, driving tremendous business growth.
- And a few weeks ago, we added another set of devices – select Android phones.
- As a result of the device implementations against the API, the Netflix API traffic has gone through the roofAveraging (today) nearly 30B requests per monthWith peak traffic at about 20,000 requests per second
- And the proportion of API requests coming to the Netflix API through the public has gone way down!The 1,000 flowers represent less than ½ a percent of the total API traffic
- Now the Netflix API has a new charter…
- Netflix engineering teams are set up in support of this charter.Internal engineering teams produce or manage content and algorithmic output (the bottom row).Different engineering and product teams are responsible for presentation layers on devices (top row).The API is responsible for delivering the content from the internal engineering teams to the device presentation layers. The API is also responsible for scaling horizontally to handle the growing load.
- Again, the Netflix API traffic has gone through the roof in this model.
- Metrics like 30B requests per month sound great, don’t they? The reality is that this number is concerning…
- In the web world, increasing request numbers mean increasing opportunity of ad impressions, which means increasing opportunity for generating revenue. And when you hit certain thresholds in impressions, CPMs start to rise, which means even more money.
- Some companies go further to generate more page views by adding things like pagination on article pages. These additional page views translate into additional ad impressions.
- But for systems that yield output that looks like this, such as APIs, ad impressions are not part of the game. As a result, the increase in requests don’t translate into more revenue. In fact, they translate into more expenses. That is, to handle more requests requires more servers, more systems-admins, a potentially different application architecture, etc.
- We are challenging ourselves to redesign the API to see if those same 30 billion requests could have been 5 billion or perhaps even less, assuming everything else remained the same. Through more targeted API designs based on what we have learned through our metrics, we will be able to reduce our API traffic as Netflix’ overall traffic grows.
- As we embark on the redesign of the API, we will plan to design it for our key audiences… the devices.
- And the redesigned API for the devices will be trickled down to other parties.
- Along the way, we plan to design the API for the audience that we want, not the audience we have. That doesn’t necessarily mean that we will implement the system for the dream audience. It just means that our designs should allow for the system to scale to the dream audience.
- For Netflix, we will be designing our server architecture, using AWS, to be highly scalable across many different areas. The API redesign will help us scale the software better to handle the scaling service more effectively and efficiently.