Deconstructing how Netflix got success thanks to a heavily personalized user experience. After the ten findings, there is a set of checklists and examples using ContentWise on how to apply the lessons to add personalization to a video service. For marketers, UI designers, multiscreen developers, TV executives and systems integrators.

1. PERSONALIZATION
10 LESSONS LEARNED from NETFLIX SUCCESS
(and how to apply them to your video services)

2. Personalization
10 Lessons Learned from Netflix’s Success
(and how to apply them to video services)

3. Greetings!
Kauser Kanji Pancrazio Auteri
Managing Editor, VOD Professional CTO, Contentwise

4. Today’s objectives
2 1
Share the findings of observing how Netflix uses
personalization as a competitive advantage in engaging
and retaining users and planning content acquisition
Show how you can connect the dots and take advantage
of those findings for your online video services

5. So what is personalization?

6. ph. Thinkstock
Not like this Personalized like this

7. It’s about the pleasure of choice and the abundance of options
It’s understanding needs and narrowing the options
to make choosing a pleasant experience

8. Personalization can go wrong

9. It’s not just high-tech
Brian & Doreen remember customers’ taste, curate and organize shelves,
plan promotions, update the display windows,
listen to requests and recommend products and let customers browse
their shop in Somerset, UK

11. Netflix
Hulu
Amazon
Home page (web)
Pure S-VOD
S-VOD Prime
Free, ad-based VOD
Upsell S-VOD Hulu+
T-VOD

12. So what’s the problem?
Broad range of user types and tastes
Fragmentation of content sources and applications: too many places to look at
Content availability can be very dynamic over time
Most UI solutions rely on drill-down and user efforts
User’s attention span and screen real estate are very limited
You name it!

13. See the opportunity?
Consumption increases
Habits formation occurs
Users feel more engaged
Things can happen when
people have a truly
personalized experience
Your service becomes a destination for unified discovery
People talk about your brand with passion
Subscribers perceive the value and the abundance of what you offer

14. Let’s start!

15. 10 lessons learned from Netflix
The information, interpretations, advice and recommendations contained in this presentation are not
endorsed in any way by Netflix and are based on information publicly shared by Netflix or its employees.

16. 10 lessons learned from Netflix
1. Set objectives and pick metrics

17. 10 lessons learned from Netflix
1. Set objectives and pick metrics
Netflix: maximize member satisfaction and
month-to-month subscription retention
Example Metrics

18. 10 lessons learned from Netflix
1. Set objectives and pick metrics
Example Metrics
Canceled subscriptions per month
Interactive sessions resulting in a playback
Played minutes per user per month
Fully watched playbacks
Interaction time before starting a playback
Returning users

19. 10 lessons learned from Netflix
2. Consider UX as mission-critical

20. 10 lessons learned from Netflix
2. Consider UX as mission-critical
secret sauce
best practices
innovative functionalities
well-tuned business rules
UX Engine Users
Content offer
Audiovisual quality
Delivery infrastructure
Editorial curation
Customer support

21. 10 lessons learned from Netflix
2. Consider UX as mission-critical
UX Engine =
Code Users
Rules
Widgets
Behaviors
Configurations
Management tools

22. 10 lessons learned from Netflix
2. Consider UX as mission-critical
Changes in UI behavior can have a dramatic impact on key metrics.
Multiscreen? Make sure behavior is consistent across devices
Pay special attention to
cross-screen consistency of
Welcome screens
Frequent user actions
User “lost” actions
Leverage the UX Engine
to control UI behavior
across all screens

23. 10 lessons learned from Netflix
3. Personalize UX as much as possible

24. 10 lessons learned from Netflix
3. Personalize UX as much as possible
At Netflix, more than 75% of views
come from some sort of
recommendation or personalized ranking
Views %
from personalized ranking
source: Netflix
0 25 50 75 100

25. 10 lessons learned from Netflix
3. Personalize UX as much as possible
User’s attention span is very limited
The first 8-12 seconds are critical
Weinreich et al. - ACM 2008
Desired outcomes
- find something to watch
- engage in some sort of exploration

26. 10 lessons learned from Netflix
3. Personalize UX as much as possible
Screen real estate is very limited too
Ideally user should
find relevant content
in the first screen

27. 10 lessons learned from Netflix
3. Personalize UX as much as possible
A
Ineffective sort criteria Effective criteria
B
Alphabetic
C
By release year
Personalized order
By ingestion order
D
Canned categories
Even “computed” lists such as
E
Most popular
Most viewed
F
Recently added
and My list!

28. 10 lessons learned from Netflix
3. Personalize UX as much as possible
There are two “folds”
Netflix personalizes in both directions
1. ranking of items in a carousel
2. ranking of carousels in the layout
2
Real estate
“above-the-fold”
1

29. 10 lessons learned from Netflix
3. Personalize UX as much as possible
Featured content
Resume play + My list
{
Most likely actions
Popular
Top picks for you
Recently added
Main genres
Pseudo-genres
Because you watched…
Friends watching…
Watch again
Displayed in a
personalized order"
"
Some of them disappear for
a while if never “touched” "
or because of A/B Testing

30. 10 lessons learned from Netflix
3. Personalize UX as much as possible
Personalized reordering may be
disorienting for some people
Indeed some Netflix users
complain about this;
but it seems to be effective
and we’ll show you
how to handle it

31. Aha!
My Aha! moment with NETFLIX

32. 10 lessons learned from Netflix
4. Understand user’s lifestyle and context

33. 10 lessons learned from Netflix
4. Understand user’s lifestyle and context
Netflix mines usage data to extract behavior patterns
Personalization may be affected by
context elements
Device type
Time of the day
Day of the week
Season of the year
User is at-home or out-of-home
Geo-location (traveling, commuting, weekend-home…)
Local weather
Popular news
Other users in close proximity (phones/wearables)

34. 10 lessons learned from Netflix
5. Use interaction data then ask for feedback
Priority on high-value usage events
Playback start/stop/resume
View asset details
Add to personal list
Other interactions
Trick-play control
Search
Sharing
Navigation paths…
Ask for feedback
5-stars
Like
Dislike
Love it!

35. 10 lessons learned from Netflix
6. Let users know how the service is adapting
to their tastes

36. 10 lessons learned from Netflix
6. Let users know how the service is adapting
to their tastes
Promote trust in the system
Encourage users to give feedback
Better personalization

37. 10 lessons learned from Netflix
6. Let users know how the service is adapting
to their tastes
Use meaningful labels
referring to past behavior
user can recognize
Because you watched Breaking Bad
Because of your interest for Time Travel
Because you loved Kill Bill Vol.1

38. 10 lessons learned from Netflix
7. Ensure metadata captures content nuances
and is consistent

39. 10 lessons learned from Netflix
7. Ensure metadata captures content nuances
and is consistent
Actors, Directors, Writers
Genres
Synopsis Release Year
Duration
Country Studio
Language
Characters Topics Themes Moods
Locations Time Periods
Keywords - Microtags
“NETFLIX QUANTUM THEORY”
A set of best practices for manual
micro-tagging of video content
Social acceptability of the lead character

40. 10 lessons learned from Netflix
7. Ensure metadata captures content nuances
and is consistent
Let users search for content you don’t have

41. 10 lessons learned from Netflix
7. Ensure metadata captures content nuances
and is consistent
With richer content metadata
you can use analytics to
understand content performance
and drive content acquisition
(or even original production)
And add meaning to user profiles

42. Movie Iron Man 3
Data from Gracenote-TMS

43. TV Series Breaking Bad
Data from Gracenote-TMS

44. 10 lessons learned from Netflix
8. Give reasons to come back often
Refresh catalog frequently - OR - Let the UX Engine do it for you (virtually)

45. 10 lessons learned from Netflix
8. Give reasons to come back often
Re-shuffle top items to periodically
change the ones above-the-fold
Items outside the first screen are
still highly relevant for the user
User perceives novelty and will
be keen to return more often

46. 10 lessons learned from Netflix
9. Run frequent UI experiments
There is no “perfect way” and there are many types of users:
experiments and adaptation seem to be the most effective ways
Identify the UI elements on the path to the key goals
Roll-out the variations and look at 2-5 metrics
Run the experiments for two weeks or until statistical validity
Design and plan experiments not to interfere with each other
Experiments consume interaction events:
make sure there is enough activity to feed all of the active variations

47. 10 lessons learned from Netflix
10. Close the loop, base decisions upon data

48. 10. Close the loop, base decisions upon data
Netflix was the only network that
said “We believe in you. We’ve run
our data, and it tells us that our
audience would watch this series.
We don’t need you to do a pilot”
Kevin Spacey, actor and producer
Listen to Kevin saying this (video)

49. 10 lessons learned from Netflix
10. Close the loop, base decisions upon data
Netflix uses analytics to heavily influence the
content acquisition policy
Netflix proved to be agile and effective in rolling out variations and
track several metrics across hundreds of client platforms
Netflix team is very disciplined on reporting UI events.
This enables full visibility in analytics and higher ROI
Yes. At Netflix they go nuts for analytics!
And they look to be right

50. 10 Lessons from Netflix - Recap
1. Set objectives, pick metrics and share them with the team
2. Consider UX as mission-critical
3. Personalize UX as much as possible
4. Understand user’s lifestyle and context
5. Use interaction data then ask for feedback
6. Let users know your service is adapting to their tastes
7. Ensure metadata captures content nuances and is consistent
8. Give reasons to come back often
9. Run frequent UI experiments
10. Close the loop and base your decisions upon data

51. Netflix solutions are applicable (and applied) at… Netflix
Other services may include S-VOD
as well as Linear TV, DVR,
Transactional VOD, Pay TV, Pay-per view,
music videos, sports highlights,
Advertising or User-generated Content…
We need a way to turn these lessons into practice
touching all the stakeholders in our projects

52. "
UIDO
A set of checklists to guide
you while introducing
personalization in your
video service

53. What you deliver How you start
User Experience Integrator Experience
UX IX
DX OX
Developer Experience Operator Experience
How you build it Tools to manage "
UIDO

54. UX User Experience What you deliver

55. UX User Experience What you deliver
Content types ✓ Movies
Aggregates ✓ Collections
✓ Series
✓ Episodes
✓ Extras
✓ Music videos
✓ Playlists
✓ News
✓ Sports events
✓ Sports highlights
✓ Scheduled programs
✓ Channels
✓ _____________________
✓ Seasons
✓ Channel bundles
✓ Movie bundles
✓ Sports Team bundles
✓ Sports League bundles
✓ __________________

56. UX User Experience What you deliver
Key UX features ✓ Manually curated collections
✓ Search results
✓ Search suggestions while you type (single/multi-type)
✓ Search refine with smart filters (facets)
✓ Similar content
✓ Personalized picks for user
✓ Critics-based feed (Rotten Tomatoes, Metacritic…)
✓ Series you watch (with next-episode)
✓ VOD bookmarking (resume playback)
✓ User’s list
✓ Predictive browsing (surfacing folders)
✓ Personalized pseudo-genres

57. Reference UI
"
Showing most of the
personalization use cases
supported by ContentWise

59. UX User Experience What you deliver
Key UX features
(cont’d)
✓ Social graph (e.g. friends, followers)
✓ Sharing actions
✓ Content can be embedded
✓ Co-watching (blended profiles)
✓ Profile explanation with content metadata
✓ User can rate content (stars, like, dislike, love, etc.)

60. UX User Experience What you deliver
For kids ✓ Parental ratings
✓ Kids mode
✓ Specialized metadata (e.g. Commonsense)
✓ Editorial curation
✓ Curation by parents
✓ Analytics for parents

61. UX User Experience What you deliver
Content sources ✓ Linear schedule (line-ups)
✓ Start-over TV system
✓ VOD Catalog
✓ Local DVR
✓ Network DVR
✓ Reverse EPG (catch-up)
✓ ______________

62. UX User Experience What you deliver
Device types ✓ Phone
✓ Tablet
✓ PC
✓ TV
✓ Watch
Access models ✓ S-VOD
✓ T-VOD
✓ Ad-VOD
✓ Free-Linear
✓ Pay-Linear
✓ PPV
Profile types ✓ Personal
✓ Household
✓ Main account powers
✓ Blended
✓ Personas templates
✓ Personal on device
Access locations ✓ At-home, OOH
✓ On-net, off-net

63. UX User Experience What you deliver
Entitlements ✓ S-VOD packages
✓ Rented movies
✓ Purchased movies
✓ Purchased seasons
✓ Purchased episodes
✓ Subscribed channels
✓ Subscribed bundles (e.g. Channel + S-VOD)
✓ ______________________

64. Explaining a recommendation
Because you liked
these other movies
Affinity between the
user’s taste and the
recommended movie
(using the tag structure)
ContentWise Reference UI

65. OX Operator Experience How to manage

66. OX Operator Experience How to manage
✓ Managing UI Elements with UX Engine
✓ Creating and updating editorial lists
✓ Generating and curating pseudo-genres
✓ Accessing analytics
✓ Content planning using analytics
✓ Managing variations and experiments for A/B Testing
✓ Understanding the impact of business rules on key metrics

68. Personalized pseudo-genres
INTENSE ACTION MOVIES
mood genre type
2000s AUSTRALIAN THRILLER MOVIES
release prod
genre type
year
country
AMERICAN DRAMA MOVIES STARRING TOM HANKS
ContentWise Reference UI

69. The magic of richer metadata
MOVIES FROM FEMALE DIRECTORS
type person role
MOVIES STARRING A ROCKSTAR
type
gender
from semantic
enrichment
looking
into actors
person role
from semantic
enrichment
ContentWise Reference UI

70. Curation of Pseudo-genres Metadata fields
considered for
labels
Status of the
pseudo-genre
Type:
Editorial
or
Computed
ContentWise Management Console

71. Driving from the UX Engine
Rendered by UI code
Configured by UX Engine
ContentWise Management Console

72. UX Engine - Personalized Order of Carousels

73. Content planning - Choosing items to retire
Find movies with a small
number of “estimated”
residual views
and are “expiring”
Automatically create a
business rule
The rule can be used in A/B Testing
to anticipate the impact of removing
these movies from the catalog.
ContentWise Management Console

74. A/B/C Testing
Biz rule #1
Biz rule #N
Variation A
Biz rule #1
Biz rule #N
Variation B
Experiment
Group A
Group B
Control Group
Results Metrics
Normal
behavior

75. Launching an experiment
This is an experiment
Treated user base
Variations

76. Monitoring experiment results
Variations Affected users Metrics
ContentWise Management Console

77. DX Developer Experience How to build

78. DX Developer Experience How to build
UI Element ✓ Carousel
✓ Group of carousel
✓ Item
✓ Item attribute
✓ __________

79. DX Developer Experience How to build
UI Events
to be reported
✓ Item displayed
✓ Item selected
✓ Item details accessed
✓ User rating submitted
✓ Explanation displayed
✓ Tuned-in
✓ Tuned-out
✓ Playback started
✓ Playback resumed
✓ Playback paused
✓ Playback completed
✓ Playback stopped
✓ Item saved to list
✓ Item removed from list
✓ Preview playback started

80. IX Integrator Experience How to start

81. IX Integrator Experience How to start
✓ Content model map
✓ Event model map
✓ User ID map
✓ Data refresh policy
✓ Bulk ingestion automation
✓ Delta updates automation
✓ Client applications map
✓ UI elements to be managed from UX Engine

82. Thank you!
For more information, please visit our website or contact us
pancrazio.kauser.kanji@vodprofessional.com auteri@contentwise.tv
Digital TV. Personalized
www.vodprofessional.com www.contentwise.tv