SlideShare a Scribd company logo
1 of 46
Download to read offline
H Y P O T H E S E S , M E T R I C S , A / B T E S T A N D D ATA A N A LY S I S
Data Driven Product Development
J U L Y / 2 0 1 8 @ F I S L
A B
Software Engineer @ GloboPlay
MSc in Computer Science @ UFCG
M a r c o R o s n e r
A B
Status
Hypotheses
Metrics
A/B test
Data analysis
Conclusions
What is the next step?
Agenda
A B
Status
Building a culture
A B
Growth time
A B
~70 AB tests in 2017
A B
Building knowledge
A B
Hypotheses
What is the problem?
Which are the solutions?
What is a good sample size?
Document it!
Metrics
One Metric That Matters
Supporting metrics
How much time is needed?
Document it!
Keep watching
A/B test
A B
What is an AB test?
AB test structure on Globo.com
A B
How to code an A/B test?
A B
H Y P O T H E S E S
If we focus on subscription when user
trying to watching videos, we believe that
we got more subscribers.
M E T R I C
Subscriptions and Conversion
D U R AT I O N
1 week
D E V I C E S
Onboarding Subscribers
Desktop Controle
S A M P L E : 8 4 %
A LT E R N AT I V E S
without-disclaimer
S A M P L E : 8 %
with-disclaimer
S A M P L E : 8 %
A LT E R N AT I V E S
H Y P O T H E S E S
If we focus on more information about the
video, we got more engagement with our
users
M E T R I C S
Video flow
D U R AT I O N
1 week
D E V I C E S
Changes on Video Stage
vs.
D E F A U L T N E W
A LT E R N AT I V E S
H Y P O T H E S E S
If we focus on show more videos on
playlist, we got more engagement with our
users
M E T R I C S
Video flow
D U R AT I O N
1 week
D E V I C E S
Changes on video playlist
D E F A U L T N E W
vs.
A LT E R N AT I V E S
Data analysis
Growth time
Notebook
Confidence index
Conclusions
Focus on user
A B
Building a Culture
A B
Building knowledge
A B
Understand your product
A B
Understand your users
A B
Understand your context
A B
What is the next step?
Data Driven Rebranding?
Questions?
talentos.globo.com
Marco Rosner
marco.rosner@corp.globo.com
@marcorosner

More Related Content

Similar to Data Driven Product Development

The Power of Bringing Your Fans Into the Marketing Mix
The Power of Bringing Your Fans Into the Marketing MixThe Power of Bringing Your Fans Into the Marketing Mix
The Power of Bringing Your Fans Into the Marketing Mix
BSI
 
Witness curriculum 1-powerpoint-setting ground rules and feedback sessions-2011
Witness curriculum 1-powerpoint-setting ground rules and feedback sessions-2011Witness curriculum 1-powerpoint-setting ground rules and feedback sessions-2011
Witness curriculum 1-powerpoint-setting ground rules and feedback sessions-2011
WITNESScurriculum
 

Similar to Data Driven Product Development (20)

Social Video - What's Next?
Social Video - What's Next?Social Video - What's Next?
Social Video - What's Next?
 
Realtime Learning: Using Triggers to Know What the ?$# is Going On
Realtime Learning: Using Triggers to Know What the ?$# is Going OnRealtime Learning: Using Triggers to Know What the ?$# is Going On
Realtime Learning: Using Triggers to Know What the ?$# is Going On
 
The Power of Bringing Your Fans Into the Marketing Mix
The Power of Bringing Your Fans Into the Marketing MixThe Power of Bringing Your Fans Into the Marketing Mix
The Power of Bringing Your Fans Into the Marketing Mix
 
Online Tool Utilization for Nonprofits
Online Tool Utilization for NonprofitsOnline Tool Utilization for Nonprofits
Online Tool Utilization for Nonprofits
 
Lean Startup Zurich- An Introduction to Lean Startup Methodology
Lean Startup Zurich- An Introduction to Lean Startup Methodology Lean Startup Zurich- An Introduction to Lean Startup Methodology
Lean Startup Zurich- An Introduction to Lean Startup Methodology
 
Applying Lessons from API Development to Healthcare Enterprise Integrations
Applying Lessons from API Development to Healthcare Enterprise IntegrationsApplying Lessons from API Development to Healthcare Enterprise Integrations
Applying Lessons from API Development to Healthcare Enterprise Integrations
 
Intro to Data Analytics with Oscar's Director of Product
 Intro to Data Analytics with Oscar's Director of Product Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product
 
Analytics in Action: How to Build Data-Informed Products
Analytics in Action: How to Build Data-Informed ProductsAnalytics in Action: How to Build Data-Informed Products
Analytics in Action: How to Build Data-Informed Products
 
Analytics in Action: How to Build Data-Informed Products
Analytics in Action: How to Build Data-Informed ProductsAnalytics in Action: How to Build Data-Informed Products
Analytics in Action: How to Build Data-Informed Products
 
Social Video Analytics
Social Video AnalyticsSocial Video Analytics
Social Video Analytics
 
Ushahidi Deployment - Output Toolbox
Ushahidi Deployment - Output ToolboxUshahidi Deployment - Output Toolbox
Ushahidi Deployment - Output Toolbox
 
Witness curriculum 1-powerpoint-setting ground rules and feedback sessions-2011
Witness curriculum 1-powerpoint-setting ground rules and feedback sessions-2011Witness curriculum 1-powerpoint-setting ground rules and feedback sessions-2011
Witness curriculum 1-powerpoint-setting ground rules and feedback sessions-2011
 
Akhil_Portfolio.pdf
Akhil_Portfolio.pdfAkhil_Portfolio.pdf
Akhil_Portfolio.pdf
 
"Testing Our Value Proposition" by Adventure Lab - Venture Lab 2012
"Testing Our Value Proposition" by Adventure Lab - Venture Lab 2012"Testing Our Value Proposition" by Adventure Lab - Venture Lab 2012
"Testing Our Value Proposition" by Adventure Lab - Venture Lab 2012
 
LSC Technology Initiative Grant Conference 2015 | Session Materials - Trust t...
LSC Technology Initiative Grant Conference 2015 | Session Materials - Trust t...LSC Technology Initiative Grant Conference 2015 | Session Materials - Trust t...
LSC Technology Initiative Grant Conference 2015 | Session Materials - Trust t...
 
Why an Indian Oxygen Plant Manufacturer is a good Choice?
Why an Indian Oxygen Plant Manufacturer is a good Choice?Why an Indian Oxygen Plant Manufacturer is a good Choice?
Why an Indian Oxygen Plant Manufacturer is a good Choice?
 
How Data Can Make You More Creative
How Data Can Make You More CreativeHow Data Can Make You More Creative
How Data Can Make You More Creative
 
YouTube Trending List: An Analysis of the Factors that Influence Video Trendi...
YouTube Trending List: An Analysis of the Factors that Influence Video Trendi...YouTube Trending List: An Analysis of the Factors that Influence Video Trendi...
YouTube Trending List: An Analysis of the Factors that Influence Video Trendi...
 
Online Live Tv: The Future of Live Linear Consumption in the Philippines
Online Live Tv: The Future of Live Linear Consumption in the Philippines Online Live Tv: The Future of Live Linear Consumption in the Philippines
Online Live Tv: The Future of Live Linear Consumption in the Philippines
 
7 Factors to Consider When Building Your Mobile Strategy
7 Factors to Consider When Building Your Mobile Strategy7 Factors to Consider When Building Your Mobile Strategy
7 Factors to Consider When Building Your Mobile Strategy
 

More from Marco Rosner

More from Marco Rosner (6)

Apresentação do SAEO na Administração Pública
Apresentação do SAEO na Administração PúblicaApresentação do SAEO na Administração Pública
Apresentação do SAEO na Administração Pública
 
[Mini-curso] Sistema de Controle de Versão
[Mini-curso] Sistema de Controle de Versão[Mini-curso] Sistema de Controle de Versão
[Mini-curso] Sistema de Controle de Versão
 
PHP Road Show - Aplicando Orientação a Objetos no SAEO
PHP Road Show - Aplicando Orientação a Objetos no SAEOPHP Road Show - Aplicando Orientação a Objetos no SAEO
PHP Road Show - Aplicando Orientação a Objetos no SAEO
 
Porque todo programador deve utilizar Sistema de Controle de Versão?
Porque todo programador deve utilizar Sistema de Controle de Versão?Porque todo programador deve utilizar Sistema de Controle de Versão?
Porque todo programador deve utilizar Sistema de Controle de Versão?
 
GuBRo - Como botar a mão na massa
GuBRo - Como botar a mão na massaGuBRo - Como botar a mão na massa
GuBRo - Como botar a mão na massa
 
Gerenciamento De Qualidade Do Projeto
Gerenciamento De Qualidade Do ProjetoGerenciamento De Qualidade Do Projeto
Gerenciamento De Qualidade Do Projeto
 

Recently uploaded

如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
0uyfyq0q4
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Valters Lauzums
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
pyhepag
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
pyhepag
 
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
acoha1
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
Amil baba
 
Abortion pills in Riyadh Saudi Arabia (+966572737505 buy cytotec
Abortion pills in Riyadh Saudi Arabia (+966572737505 buy cytotecAbortion pills in Riyadh Saudi Arabia (+966572737505 buy cytotec
Abortion pills in Riyadh Saudi Arabia (+966572737505 buy cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
dq9vz1isj
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理
pyhepag
 

Recently uploaded (20)

How I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonHow I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prison
 
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
 
Seven tools of quality control.slideshare
Seven tools of quality control.slideshareSeven tools of quality control.slideshare
Seven tools of quality control.slideshare
 
AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdf
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
 
Aggregations - The Elasticsearch "GROUP BY"
Aggregations - The Elasticsearch "GROUP BY"Aggregations - The Elasticsearch "GROUP BY"
Aggregations - The Elasticsearch "GROUP BY"
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
 
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
 
What is Insertion Sort. Its basic information
What is Insertion Sort. Its basic informationWhat is Insertion Sort. Its basic information
What is Insertion Sort. Its basic information
 
How to Transform Clinical Trial Management with Advanced Data Analytics
How to Transform Clinical Trial Management with Advanced Data AnalyticsHow to Transform Clinical Trial Management with Advanced Data Analytics
How to Transform Clinical Trial Management with Advanced Data Analytics
 
Artificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfArtificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdf
 
Abortion pills in Riyadh Saudi Arabia (+966572737505 buy cytotec
Abortion pills in Riyadh Saudi Arabia (+966572737505 buy cytotecAbortion pills in Riyadh Saudi Arabia (+966572737505 buy cytotec
Abortion pills in Riyadh Saudi Arabia (+966572737505 buy cytotec
 
Genuine love spell caster )! ,+27834335081) Ex lover back permanently in At...
Genuine love spell caster )! ,+27834335081)   Ex lover back permanently in At...Genuine love spell caster )! ,+27834335081)   Ex lover back permanently in At...
Genuine love spell caster )! ,+27834335081) Ex lover back permanently in At...
 
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理
 
The Significance of Transliteration Enhancing
The Significance of Transliteration EnhancingThe Significance of Transliteration Enhancing
The Significance of Transliteration Enhancing
 
123.docx. .
123.docx.                                 .123.docx.                                 .
123.docx. .
 

Data Driven Product Development