Data Driven Growth - Amplitude London Product Analytics Summit

📈 🚀
Andy Young
Amplitude Product Analytics Summit 2017, London
Product analytics tips learnt from 50+ companies
Data-Driven Growth

Andy Young - @andyy - http://bit.ly/analytics-summit-andy
Hi, I’m Andy!
@andyy
#fb
Photo by Dan Taylor - dan@heisenbergmedia.com

Agenda
Why analytics?
Deconstructing the modern analytics stack
Lies, damn lies, and tech metrics.. 
- analytics in the real world

Andy Young - @andyy - http://bit.ly/analytics-summit-andy 🤔
Why analytics?

Don’t start with the data

Common Analytics FAILS
• Drowning in data
• Not identifying  
key questions to answer
• Not starting with 
clear hypotheses

We need to know how we’re doing

• Overall business/product
• Individual features, experiments, customer segments

• Overall business/product
• Individual features, experiments, customer segments
• What is working?
• What is not?
• Where to focus for improvement?

Analytics Pros..
• Start with a hypothesis
• Identify, collect + analyse necessary relevant data
• Conclude & action
• Iterate, revise

Analytics for Growth
📈

Growth?
•More users
•More engagement
•More revenue

Analytics techniques
• Early stage: 
- innovation accounting (Lean Startup) 
- cohort analysis

Analytics techniques
• Early stage: 
- innovation accounting (Lean Startup) 
- cohort analysis
• Growth stage: 
- growth accounting 
- feature lifecycle analysis 
- user segmentation

Early stage questions

Are we getting traction?

“Traction is the
rate
at which
monetizable value
is extracted
from customers”

Innovation
accounting?

Cohort analysis?

Growth stage questions

Are we growing?

Growth Accounting

Growth Accounting
👌👉 http://bit.ly/social-capital-growth-accounting

Funnel performance?

As product folks 
we need to understand
the full user journey

Customer segmentation
For different groups of customers:
• Where do they come from?
• How do they behave?

Users with different personas & needs 
will have different behaviours
Different cohorts will have  
different experiences of your product
Different users will have been exposed to 
different A/B tests

These are all properties of your users
UTM tags: source, medium, campaign, terms
Landing page
Signup time
A/B test buckets
Referrer
Viral source

Most experiments FAIL

Most experiments fail
..but most every failed experiment will
contain partial successes
• Plan in advance: how will we know 
at which point in the user journey  
an experiment failed?
• Do things that don’t scale

Pulling this all together
Why analytics?

What to track?
• Discovery / Acquisition
• Activation
• Engagement
• Conversion / Purchase
• Retention
• Referral

So many platforms

So many platforms
¯_( )_/¯

What functionality
do you need?

What functionality do you need?
• Session/Pageview Analytics 
Google Analytics, Chartbeat

• User/Event-based analytics 
Amplitude, Mixpanel, Heap, Kissmetrics, Localytics

• Mobile deep linking and attribution 
Branch, Adjust, AppsFlyer, Tune, Kochava

• Mobile-specific 
Flurry, Swrve, Leanplum

• A/B testing 
Optimizely, WVO, Google Content Experiments, StoreMaven

• A/B testing 
• Querying & Charting 
Mode, Periscope, Tableau, RJMetrics, Statsbot

• A/B testing 
• Dashboards 
Geckoboard, Klipfolio

• A/B testing 
• Dashboards 
• Audience demographics, interests & rankings 
Quantcast, Comscore, Alexa, SimilarWeb

• A/B testing 
• Dashboards 
• Marketing Automation, CRM, Email & Push Notifications 
Marketo, Hubspot, AppBoy, Kahuna

• A/B testing 
• Dashboards 
• Platforms 
Google Firebase, AWS Mobile Analytics

• A/B testing 
• Dashboards 
• Platforms 
Google Firebase, AWS Mobile Analytics
😱

How to pick?
• What functionality do you need?
• Who will be using it? Devs, data scientists, product folks, marketing people?
• How do you want to use it? Analysis, reporting, dashboards..
• Which platforms? Web, mobile, server?
• Which other tools/data do you need to integrate with?
• What data volumes? Data points, events per second, data points per month.
• Budget?

How to pick?
Things to be wary of:
• Data lock-in, future portability
• Building your own

Tools are not a panacea

Biggest single factor?

Biggest single factor?
Quality of your  
tracking implementation
- Coverage/depth
- Accuracy/lack of bugs

Andy Young - @andyy - andy@apexa.co.uk
Use a tag manager

Data collection: Segment.com

👉 http://sitehound.co
Free code!

Dashboards?

What makes a
good dashboard?

👉 http://bit.ly/kpi-sheet

Growth model spreadsheet
• Review weekly - zoom out vs. zoom in
• Share with whole team
• Use to predict and prioritise
• What to look at daily vs. weekly vs. monthly
👉 http://bit.ly/kpi-sheet

Metrics are people too
Key growth metrics should be measured in 
unique people
or % conversion

Joining the dots
Plugging all these tools and platforms together

Initially, manual is OK!

Initially, manual is OK!
• Forcing function to ensure we:
• learn what works
• understand the data
• need the data
• Paste weekly into a Google Sheet/Excel

Automate over time
to reduce effort

Plumbing for automation
• Zapier, Tray.io
• Supermetrics, Blockspring for Google Sheets/Excel
• Segment.com
• Own database/data warehousing - AWS RDS/Redshift
• Custom queries - Tableau, Periscope etc

Why analytics?

The numbers
never add up

The 5 stages of 
(Analytics)  
Grief

The 5 stages of Analytics Grief
• Denial

FB ad clicks
vs.
GA visits

• Denial
• Anger

Analytics vs. own database

• Denial
• Anger
• Bargaining

Measure
Twice:
Compare
data across
different tools
https://jeffpeachey.com/2010/12/02/measure-twice-cut-once-woodcut-for-sale/

• Denial
• Anger
• Bargaining
• Depression

Where did
it all go
wrong?

Where did
it all go
wrong?
• The data is bad
• Our definitions are wrong

The numbers never add up
• Absolute truths
• Signups, Transactional data, UGC, data from
our own DB
• Lossy/noisy measurements
• Client-side tracking: Amplitude, GA, etc
• Nuanced definitions
• Uniques vs. totals, funnel conversion rates

Even if the numbers do add up,
they are still often misleading

Data discrepancies
Amplitude vs. Google Analytics
• Definition of unique user 
- cookie rules
• Ad/tag blockers

Data discrepancies
• Page did not finish loading 
- e.g. tag manager, or user gave up & hit
back button
• Cross-browser JS bugs

• Denial
• Anger
• Bargaining
• Depression
• Acceptance

3-5% discrepancies between
platforms?
Roll with it.

Large (>5%) difference?
Debug/analyse to figure out why

Takeaways?

Secrets to successful analytics
• Start with your questions / hypotheses
• Be precise about what to measure, and why
• Plan in advance, simplify
• Automate required work for weekly/monthly
calculations
• Document all analysis with simple bullet-points

The numbers tell us What
They don’t tell us Why

Thanks, & good luck!
@andyy // andy@apexa.co.uk
Slides: bit.ly/analytics-summit-andy

References & further reading
• My more in-depth analytics presentations: Slideshare, YouTube
• KPI google sheet
• SiteHound website tracking javascript library
• UTM Tagging Guide
• Lean Analytics
• Social Capital blog posts: - Accounting for User Growth

Data Driven Growth - Amplitude London Product Analytics Summit

Recommended

Recommended

More Related Content

What's hot

What's hot (11)

Similar to Data Driven Growth - Amplitude London Product Analytics Summit

Similar to Data Driven Growth - Amplitude London Product Analytics Summit (20)

Recently uploaded

Recently uploaded (20)

Data Driven Growth - Amplitude London Product Analytics Summit