Who, how much, and how often? The difficult questions answered around your A/B testing strategy

The Past, Present and
Future of Website
Optimisation
In association with Qubit
Linus Gregoriadis
Research Director
@LinusGreg

Introduction and Methodology
The Business Case
Foundations and Resourcing
Moving Towards Personalisation
| 6 | The Past, Present and Future of Website Optimisation

The Business Case

Aims of the report
● To find out how companies are approaching the issue of
testing and optimisation.
● To identify the barriers to optimisation.
− Technological
− Cultural
● To understand the processes and approaches used.
● To highlight any potential future challenges.

The Business Case

Photo credit: 123dan321 on stock.xchng
Why test anyway?

Testing is key to improving website
performance
Source: Conversion Rate Optimisation Report 2014

Testing is key to improving website
performance
● Companies seeing significant increases in sales
typically run more A/B and multivariate tests

“If it is going to fail, fail fast…
“We see a positive benefit from around a
third of tests.”
Chris Howard, Head of Digital, Shop Direct

Case Study: Topshop
● Wanted to improve US
experience
● Tested using size formats
more familiar to North
American shoppers
● Measured not on
conversion but revenue
optimisation
● Increased revenue by
4.5% with 99% statistical
significance

Photo credit: 123dan321 on stock.xchng
What to test?
What to aim for?

“The onus is on optimising the funnel and we
can’t do it all at once. We’ve started having
grown-up conversations about what is the
best point to focus optimisation on.”
Daniel Sale, Head of Digital Marketing, Investec
Private Bank

Advanced Website Optimisation is yet to
get over the adoption curve…

The Business Case

How much to test?

“We’ve learned what degrees of success you
can have. You can chip away and only move
the dial so far unless you are doing
something significantly wrong.”
Paul Wishman, Group Ecommerce Director, LV

Size matters

“Realistically we can run three to four tests a
month simultaneously… we never get the
volume to the extent of retail banks for
example. We need statistically significant
samples and that takes time.”
Daniel Sale, Head of Digital Marketing, Investec
Private Bank

“Apart from choosing the right tools, most of
the obstacles were related to internal culture
and getting buy-in from internal
stakeholders.”
David Paice, Group Ecommerce and Admissions
Director, Merlin Entertainments

“Testing has been an internal
communications piece and especially
because A/B testing involves so many
different teams.”
Rachel Waller, Senior Digital Marketing and
Social Manager, Farfetch

Decisions on
technology

• Bought-in software,
either off the shelf or
customised?
• Built in-house?
• Managed services
(when needs are
constantly
changing)?

Can your vendor
grow with you?

The Business Case

“Personalisation is the most exciting
challenge and the hardest. There’s so much
we can do but it’s finding out what you
should and what will add value.”
Rachel Waller, Senior Digital Marketing and
Social Manager, Farfetch

Case Study: Farfetch
● Farfetch hypothesised that
people convert more when
they understand the
business model
● Segment of users: new
visitors with over 10 page
views in their session
● A message layer slides in,
prompting users to read
FAQs
● Result: 17% increase in
conversion

93% of companies saw
an uplift in conversion rates
since implementing
personalisation.

Ensure incremental
gain

“Testing by revenue is critical to our
business. Conversion tells only part of the
story.”
Simon Pritchard, Group Digital Director, Arcadia

One to One?

“It’s the nature of fashion for consumers
largely to behave like the herd in many ways.
Perhaps one-to-one conversion isn’t
necessary. If you can identify groups of
people, that’s good enough.”
Stuart McMillan, Deputy Head of Ecommerce,
Schuh

Case Study: DFS
● Hypothesised: European
customers may need more
information when
shopping on UK websites
● Targeted users from
outside the UK with a layer
informing them that DFS
shipped to their country
● Result: 23% uplift in
conversions for segment

Over time the majority will adopt best
practice

Final Takeaways
● Testing has seen a surge of
growth but barriers still
remain.
● People elements are often as
significant as technological
ones.
● A pragmatic yet continual
approach is most likely to
pay future dividends.

PEOPLE, PROCESS, TECHNOLOGY
NAOSHI YAMAUCHI, CPO
@Yamalytics

Brooks Bell
The premier firm focused exclusively on
enterprise-level A/B testing, optimization and
personalization services.
SOME OF OUR CLIENTS

TALENT, CULTURE, PROCESS ISSUES CAUSING
‘UTILIZATION GAP’
Optimization tools increasing in sophistication
• Tracking across platforms
• Personalization
• Mobile
• More data!
42
Capabilities

TALENT, CULTURE, PROCESS ISSUES CAUSING
‘UTILIZATION GAP’
Optimization tools increasing in sophistication
• Tracking across platforms
• Personalization
• Mobile
• More data!
43
Insufficient Team & Talent
• Business savvy analysts
• Technical talent
• Data savvy executive influencers
Capabilities
Broken Process
• Lack of standards
• Minimal documentation
Resistant Culture
• Gut-driven decision
making
• Minimal training & sharing
UTILIZATION
GAP
Use
DRIVING
ISSUES

44
Utilization ranks 20th out
of 24 total categories in
Top
Scoring
Assessment Brooks Bell Strategic Assessment
Categories
Worst
Scoring
24

So, What can you do?
4 TIPS ON HOW TO CLOSE THE UTLILZATION
GAP

46
ASK YOURSELF:
“Do we really have
the winning team?”
If the answer is no,
you’re not alone ☺.
tip1

OPTIMIZATION TEAM
Strategy
Design
Engineering
Project
Management
Analytics
Creative

EVANGELIZE &
BUILD CULTURE
GET
INFLUENCERS
RECOGNIZED IN
TEST WINS
tip2

WHO IS BROOKS BELL?
WE RING A GONG EVERY TIME WE LAUNCH A TEST

CREATE A REPEATABLE PROCESS
Standardize and then document
campaign, dev, QA, test archives
YOU CANNOT SCALE AND
MAINTAIN MOMENTUM
WITHOUT
PROCESS & STRUCTURE
tip3

51
THE
STORY
OF TESTING
AT BRAND X
2011 2012 2013 2014
COMPLEXITY
DEMAND
TEAM SIZE
# OF TESTS
PROCESS
CULTURE
DOWNLOAD! Campaign QA Checklist – bbi.bz/qa-2014

tip4
START TARGETING SEGMENTS FOR TESTING.
IT WILL MAKE IDEATION MORE EFFECTIVE.
DOWNLOAD! Campaign QA Checklist – bbi.bz/qa-2014

Case Study: New v. Returning
Segment: New customers
Goal: Increase orders
Issue: Confused as to what
the different fits mean
CONTROL CHALLENGER

54
TIP 1: Ask yourself “Do we really have a winning team?”
TIP 2: Get influencers recognized in test wins
TIP 3: Create a repeatable process
TIP 4: Start targeting segments for testing

Who, how much, and how often? The difficult questions answered around your A/B testing strategy

Who, how much, and how often? The difficult questions answered around your A/B testing strategy

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Who, how much, and how often? The difficult questions answered around your A/B testing strategy

Similar to Who, how much, and how often? The difficult questions answered around your A/B testing strategy (20)

More from Qubit

More from Qubit (20)

Recently uploaded

Recently uploaded (20)

Who, how much, and how often? The difficult questions answered around your A/B testing strategy