5. 5
5
https://twitter.com/Thomasbcn/status/690109233510105089"We have increased install conversions
by more than 25% testing content for our app stores”
Maria Waters (Kabam)
“If we can improve the app store conversion,
we actually reduce the load on our marketing team
and significantly reduce the budget that we spend on user acquisition”
Rajeev Girdhar (Rovio)
“I changed my icon to an icon with a unicorn.
My daily downloads as well as daily revenue doubled!”
Per Haglund (Super Puzzle)
Source: Splitmetrics, VB, StoreMaven
Why?
@Thomasbcn #APSLondon
7. 7
7
Gut choices vs testing
“Design by committee produces disastrous results”
Michael D'Ulisse (TapTapTap)
“A/B testing revealed that the images the team preferred reduced the
conversion rate of installs by 15%”
Fat Fish Games (VB)
“I’ve seen pageview to download conversion rates
rise double digit percentages through icon optimization”
Wilhelm That (Rovio)
@Thomasbcn #APSLondon
9. 9
9
A/B testing iOS Search Ads
“The top app achieved approximately the same amount of clicks
on the ‘Get’ button regardless of whether it was a ‘search ad’
or a regular search result”
“Users don’t see the difference between
sponsored and organic listings”
@Thomasbcn #APSLondon
10. 10
10
How to? Apple AppStore
• No official tool
• Send paid & owned traffic
to an AppStore replica (web)
• Providers:
Build your own? (Soundcloud)
youtube.com/watch?v=v8MXAzYPKFY
@Thomasbcn #APSLondon
12. 12
12
Apple AppStore A/B testing caveats…
• Misleading users
• Organic traffic excluded
• Added costs: tool + dropoff
“You’ll be losing some users who’ve decided to download your app”
Nick Kurat (TestNest)
“There was a big drop-off which made the cost to acquire each
sample user much higher than expected:
greater than $10 on Google Play.”
Ilya Shereshevsky (DeNA)
Source: TestNest, AppAnnie @Thomasbcn #APSLondon
13. 13
13
How to? Google Play Store
Play Store Experiments
• Built-in (Dev. Console)
• Includes organic traffic
• Test visuals & texts
• Up to 5 localized experiments at a time
• FREE
@Thomasbcn #APSLondon
18. 18
18
Google Play Store A/B testing limits…
• Android & iOS are different
• No testing on title
• Traffic split uncontrolled & undisclosed
• No user behaviour details
in app / on bottom line
on page
• Unreliable statistical relevance…
Read more: GP Experiments limits acc. to
@Thomasbcn #APSLondon
22. 22
22
The issue with GP Experiment:
Undisclosed statistical significance
@Thomasbcn #APSLondon
23. 23
23
A/B testing listing caveats…
• Most A/B tests fail!
• Closed variants bring little learnings
• Volume required for significance can be high
• Improvements curve
• Bottom line impact?
• Bias/externalities
“Only 1 out of 8 A/B tests have driven significant change”
@noahkagan
“We waste our time testing variants
without any real meaningful differences”
@justinrmegahan
Read more: ConversionXL, Formisimo, Groove @Thomasbcn #APSLondon
24. 24
24
@Thomasbcn #APSLondon
… Believe your guts
… Design by committee
… Start with close variants
… Assume iOS & Android behave the same
… Mimick your competition
… Conclude on early data
… Believe minor uplifts
… Believe GP Experiments stats relevance
… Test & iterate
… Make bold assumptions
… Focus on visuals
… Localize your tests
… Counter-test results
… Double-check with tracked sources
… Challenge the tools
… Challenge yourself