Culture of Optimization

PRESENTED BY:
ERIC HANSER – DIGITAL OPERATIVE
Culture of Optimization
Ecommerce A/B and Multivariate Testing Primer

digitaloperative.com Tweet us @digitaloperativ

THESE ARTICLES WILL NOT
GIVE YOU A STRATEGY
Some of these tests may be effective, and some may not.
Probably more of the latter.
Every site is unique and has unique users with unique
problems.
Without more concrete hypotheses, a failed test leads to a dead
end.

NUMBER 1
Getting your site into shape is
just like getting yourself into
shape.
There is no magic pill or simple
solution. It is a lifestyle change.

OK. SO WHERE DO YOU
START?

LET’S TAKE ONE STEP BACK.
WHAT DO WE MEAN BY
“TESTING”

SPLIT TESTING: SIMPLE
IDEA, LOTS OF
POSSIBILITIES
A split test is a controlled, randomized experiment run on your
live site.
Users to your site are split up and shown different versions of
the site.
The users’ progress is tracked to determine which version will
be most successful.

SPLIT TESTING: PLATFORMS

CRO = CONVERSION RATE
OPTIMIZATION

WHAT HAPPENS IF YOU
IMPROVE CONVERSION
RATE, BUT AOV GOES
DOWN?

THE TERM CRO CAN BE
SOMEWHAT MISLEADING
A “conversion” technically refers to a a user completed any
desired action.
In ecommerce, the word “conversion” is almost always used in
reference to making a purchase.

THE TERM CRO CAN BE
SOMEWHAT MISLEADING
Split testing can be used effectively to help with everything from
branding to design to marketing, etc.
By using the term “CRO,” some departments or colleagues
might not see immediately that these tools and processes are
relevant and helpful to the work they do.

NUMBER 2
In building a culture of
optimization, it may be helpful to
limit how often you use the term
“CRO.”

TO AVOID CONFUSION,
USE THE TERM
OPTIMIZATION & TESTING

WHAT ARE THE KINDS OF
TESTS WE’RE TALKING
ABOUT?

A/B TESTS:
SIMPLEST (AND RAREST)
A/B Testing simply means testing one element on a page. Each
option is called a version or a variation.
Not limited to 2 variations. Sometimes called A/B/C or A/B/N
testing.
Only a single element is tested: color, placement, wording,
picture, etc.
If more than one of these elements change, it is no longer an
A/B test.

TYPES OF TESTS: A/B
TESTING

MULTIVARIATE TESTING:
A BIT MORE TIME
When more than one element on the page is being tested.
Each possible combination of elements should be tested.
With more variations, it can take longer to get statistically
significant results (more on this later).
All variations are still on a single page.

TYPES OF TESTS:
MULTIVARIATE TESTING

MULTI-PAGE TESTING:
USERS SEE MORE THAN
ONE PAGE
If an element appears on more than one page, you must test
the change on each page.
Also often called “Flow Testing.”
Multi-page testing may be either A/B or Multivariate.

TYPES OF TESTS:
MULTI-PAGE TESTING

SPLIT TESTING TYPES
A/B Tests – A single-page test with a single element changed
Multivariate Tests – A single-page test with more than one
element changed.
Multi-page Tests – A test that shows a change across more than
one page in the user flow.

ALL OF THESE TEST
RANDOMLY SPLIT USERS
INTO GROUPS.

TARGETING: ALL USERS ARE
NOT THE SAME.
Optimizing your site for all users is important, but speaking
directly to specific segments is even more effective.
Many of the same tools that are used to run split testing can
also be used to target specific audiences and optimize the site
for them.

NOT ALWAYS A TEST:
SEGMENTING AND
TARGETING

NUMBER 3
Results from testing can lead to
targeting.
When looking at results of your
test, you may find that certain
segments should be targeted
differently.

SO, WHERE DO YOU BEGIN?

IT IS TEMPTING TO
JUMP RIGHT IN
It may seem that there are things “wrong” with your site or that
you can make tests based on “best practice.”
Tests have “winners” and “losers” - there will be scrutiny.
The more information you have, the better chance you have of
making wins early on.

NUMBER 4
Don’t start with testing!
If your first few tests are flops,
confidence in the whole idea of
split testing could be
compromised.

START WITH LEARNING THE
“WHATS” AND “WHYS” OF
YOUR SITE.

FIND THE “WHAT”
DATA, DATA, DATA!

FIND THE “WHAT”
What are users doing or not doing? What do they like and/or
dislike?
Learn all you can from analytics. Proper set up and tracking
could take time.
Ask questions like:
What are the biggest drop-off and exit points?
What pages are most successful?
What are the most optimal site flows? What are the least
optimal?

FIND THE “WHY”
USABILITY TESTING

FIND THE “WHY”
With data collected from analytics, develop paths or a series of
tasks for users to complete in a usability test.
Several tools are available to either conduct in-person or
remote usability testing. Usertesting.com is one of the most
popular.
Often, unexpected and surprising results surface that you would
have not necessarily seen in the analytics alone.

YOU KNOW THE PROBLEMS.
NOW, MAKE A HYPOTHESIS.

START WITH THE
HYPOTHESIS – THEN
DEVELOP THE TEST
When making a test, base your hypothesis on the problem.
“If I change X, then Y should improve.”
Never make a test and THEN ask the question, “how will I know
which variation won?”
Simply looking at overall ecommerce conversion will likely be
inconclusive.

WHAT & WHY
30% of traffic direct to
product pages
63% bounce rate
“Best Practice”
recommendation to
increase the add-to-cart
button size and the free
shipping message

WHAT & WHY
Users had no validation
that the product was right
for their specific
condition.
Users had no clear way
to find the right brace for
them.

GETTING RESULTS
STATISTICS, BLEH!

GETTING RESULTS –
DETERMINING THE WINNER!
Two primary metrics:
Percentage Increase = How much a variation improved
over original
Confidence Interval = How confident we are that the
improvement is not due to chance
A 10% improvement with an 85% confidence interval basically
means that there is an 85% chance that the test will result in a
10% improvement.
95% confidence interval = Statistically Significant

GETTING THE RESULTS
STATISTICAL SIGNIFICANCE
10%
Improvement
85% Confidence
Interval
10%
Improvement
95%
Confidence
Interval
15%
Improvement
75%
Confidence
Interval
Original

WIN, LOSE, OR DRAW
Optimization is ongoing and iterative.
Ultimately, if a test is based on data and usability, we’re always
learning and improving.

NUMBER 5
If your testing is always based on
data, every test will be a win.
If your tests aren’t based on data,
even wins may be losses.

WINS & LOSSES – WHEN A
WIN IS NOT A WIN.
If a test is implemented without a clear hypothesis and clear
goals, a win can lead to a dead end.
If you don’t have a clear idea as to WHY a variation won, we’ve
got nowhere to go.

WHEN A WIN IS NOT A WIN

WINS AND LOSSES – WHEN A
LOSS IS NOT A LOSS
Similarly, we can learn a lot when a variation underperforms if
the hypothesis was based on data.
With new information in mind, we can look again at the problem
with additional data to find other solutions.

WHEN A LOSS IS NOT A LOSS
digitaloperative.com

digitaloperative.com
WHEN A LOSS IS NOT A LOSS

WINS & LOSSES – WHEN A
DRAW IS A WIN
Occasionally, a test goal will be something less “tangible” than
an increase in conversions.
Some tests’ goals, for example, will be to bring consistency to a
cluttered site, make copy more on brand, or to adapt to
changing business requirements.

WINS AND LOSSES
WHEN A DRAW IS A WIN

MAKE A PLAN – PRIORITIZE!
Not everything can be tested at once.
The P.I.E prioritization process helps put all possible tests in a
digestible order.
P = Potential / How big is the problem?
I = Importance / How valuable is the traffic?
E = Ease / How difficult will the test be to implement?

MAKE A PLAN
PRIORITIZE!
WEBSITE TESTING Prioritization
Test Name HYPOTHESIS Potential Importance Ease PIE AVG PIE Notes
Product Page - Short
Product Description
By giving users feedback
related to their path (filters,
categories, search terms)
they will be more confident in
purchasing 8 9 7 8
High Traffic, High Bounce
Rate, text-only
implementation, but need use
of audiences tool
Cart / Global Nav - View
Cart
Design and placement of
"view cart" CTA will bring
more people back into the
purchasing funnel if they
leave the cart 6 8 9 7.666666667
Valuable traffic, but most
users are directed to the
shopping cart. Easy
implementation
Category -
Subcategory/Filter Design
By providing clarity about the
differentiation between
subcateory links and filters,
potential UX issues will be
mitigated 8 8 7 7.666666667
Noted problem/issue in
usability. Use of filters shows
intent, and therefor
important.
Checkout - Security
Messaging
Overstating Security may
actually be harming
conversion. Possibly finding a
more optimal amount of
messaging could improve it 5 9 9 7.666666667
No known issue, but hightly
valuable traffic, and
extremely easy test to
implement.
Product Page - Find Other
Products
Allowing users on a PDP to
easily click off to other similar
products if it not appropriate
for their needs, we decrease
bounce rates 8 7 7 7.333333333
Lots of traffic, high bounce
rate, but the feature would be
new, and therefore
importancei is measured
Global Nav - Elements and
Design
Users perform better when
accessing injury categories
as opposed to body type
cateogirs 8 8 6 7.333333333
As a global element,
importance and potential is
high. Implementation will be
slightly more than others.

NUMBER 6
Prioritization not only keeps you
organized, it helps your
organization understand the
breadth and importance of
optimization and testing overall.

CULTURE OF OPTIMIZATION
WEBSITE FITNESS TIPS
There is no magic pill – It’s a lifestyle change.
The term “CRO” can be confusing – Use “Optimization &
Testing.”
Test then target – Some segments may need to be targeted
differently.
Don’t start with testing – Gather data (“whats” and “whys”) first.
Base tests on solid data and hypotheses – All tests can be
wins.

QUESTIONS?
ERIC@DIGITALOPERATIVE.COM
WWW.DIGITALOPERATIVE.COM

PLEASE USE THE OR APP TO RATE THIS SESSION
Thank you for attending!

Culture of Optimization

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Culture of Optimization

Similar to Culture of Optimization (20)

Recently uploaded

Recently uploaded (20)

Culture of Optimization

Editor's Notes