“I Added +$3M to Client Revenue With CRO & UX”

IMPROVE
YOUR
CRO
PROGRAM
A 30MIN WEBINAR + 10MINS Q&A
VWO Webinar 2022
I Added +$3M to Client
Revenue With CRO & UX
A Webinar by Chris Marsh

CHRIS MARSH
CRO FOR ECOMMERCE
10 YRS IN UX/DATA, 5 YRS IN & CRO
Brands I've worked with - CRO, UX and Analytics…

2 CRO PROGRAMS
THAT FAILED
01.
THRIVING CRO
PROGRAMS, +$3M
02.
MAKE CRO EASIER
& LONG-TERM
03.
6 PRACTICAL TIPS
04.
Q&A
VWO Webinar 2022
I Added +$3M to Client
Revenue With CRO & UX
A Webinar by Chris Marsh
What to look out for and avoid
6 key take-aways
A new framework to guide you
Wrap-up
A 30MIN WEBINAR + 10MINS Q&A

2 CRO
PROGRAMS
THAT FAILED
01.

“WE LOVE YOUR WORK,
BUT WE JUST DON'T HAVE
THE DEV RESOURCES”

MAKE SMALL
BETS (TESTS)
TO BUILD MOMENTUM. LEARN. ITERATE.
•
• Avoid large risky tests…
Can you learn more easily?
• If you do run large tests, run small ones at the same time.

SMALL BETS WASN'T MY IDEA.
GREAT AUTHORS BACK IT UP!
"To maximize your chances of success you should deploy
small, concrete experiments that return concrete feedback"

“THE OWNERS ARE
OLD-FASHIONED, AND
DON'T SEE THE VALUE”

KEEP SHOWING
VALUE
BEAT. THE. VALUE. DRUM.
PR your CRO Program
Highlight the value every quarter
Keep it short and simple

PR your CRO Program - An Example
EXAMPLE
BEAT. THE. VALUE. DRUM.

BONUS
OPPORTUNITY
TO GROW YOUR
PROGRAM
Associate success and value
of the program to the resources
Thanks to

Rommil Santiago

for this tip

THAT WAS 2
CRO PROGRAMS
KILLERS:
1) LOST MOMENTUM
2) NOT SHOWING
VALUE REGULARLY

THRIVING CRO
PROGRAMS, +$3M
02.
6 key take-aways

1. TEST ONE CONCEPT
CHANGE, AND YOU’LL
NEVER LOSE
You'll learn if your solution changed behavior.

If it resonated with the audience.
Test to learn - a variation should answer one question

2. MOVE ON OR ITERATE?
Always deep dive to gain insights and consider new test ideas.

Sometimes a test looks like a loser, but wins on Mobile.

Test Idea #1 Test Idea #2 Test Idea #3
Test Idea #3.1 Test Idea #3.2
Test Idea #4
CRO Roadmaps shouldn't be long and rigid. Be very flexible!
It can be worth iterating after…
- a win: can you do it elsewhere or better?
- a loss: try a new solution?

3. WE HAVE TO PUT IN
THE HEAVY LIFTING
CRO Research
• User Research
• Previous Test Learnings 📖
This will boost your win-rates by 2x
eg. from 10% winning tests to 20%+
^ GREAT data source!

4. LOG LEARNINGS
Past learnings help you move in the right direction
Use VWO, or AirTable to maintain your CRO program.

Get the CRO & Research Hub template here: bit.ly/CROHUB-DCX
Thanks to Tim Altman for his AirTable inspirations - which led to this evolution I use.

Tim Run's ExceptionalWeb - a network of senior freelancers, myself included.

BONUS - TAG TESTS
WITH A PURPOSE
To see patterns in what works best, or missed opportunity.

Add 1-2 tags per test
.
Recommended reading - Purpose Testing,

by Growth Rock Agency

5. QA - CHECK YOUR
TEST FOR BUGS!

On a redirect test: On the new page, make sure the dev adds the rel canonical tag into HTML HEAD

- to point to the original page. Otherwise you may get an SEO Nightmare.
5. QA - CHECK YOUR
TEST FOR BUGS!
Check the new experience on an old iPhone, an old Android, on Mac + Windows

Ideally on real devices. Otherwise on BrowserStack.

Takes 0.5-2 hours. Get some good music on, and blitz through this boring but essential task! 🎶

“perhaps the most important single law in the whole of data analysis.”
AND LASTLY…
THE MOST IMPORTANT LAW
IN DATA ANALYSIS

6. TWYMAN'S LAW
Variation conversions were up +130% uplift over 5 days
"Any figure that looks interesting or different is usually wrong"
REMEMBER TWYMAN'S LAW
IT SHOWS UP ALL THE TIME!

A FRAMEWORK TO
MAKE CRO EASIER
& LONG-TERM
03.
PLUS A REAL TEST SHOWING THIS IN ACTION

WHAT'S IMPORTANT
USE CRO RESEACH DATA!
GET A HIGHER CHANCE TO WIN
TEST TO LEARN!
CRO PROGRAM IMPROVES OVER TIME
Surveys / Polls
User Testing
Heatmaps
Voice of Customer
Expert Review
Google Analytics
e.t.c
WHY?
SO YOUR WIN RATE GROWS
FROM 10% TO 20%+

This knowledge helps you…
TEST TO LEARN.
WHY DOES THIS WORK?
- move in the right direction
- learn
- iterate
- and test in areas that move the needle
Highly Recommended deep dive - click to watch
For more Test-to-Learn experimentation tips

By Shiva

LET’S SEE THE
FRAMEWORK 👀
WHICH GUIDES YOU
TO TEST & LEARN

PROBLEM
Customer problem,
backed by data
P
SOLUTION
With input from the
wider team
S
Q
QUESTION(S)
What do you want to know?
PQ FRAMEWORK FOR CRO
Optional - add a hypothesis if you want to state your reasoning:
If (we make this change.....)

then (KPI X will increase.....)

due to (theory/reason...........)
M
MEASURE
Primary KPI,
Secondary KPIs
This intuitive framework will ensure you Test to Learn, and you'll get to more wins than testing random ideas.
Steps: Add these 4 items to your Test Ideas
Created By
Important:
Test 1 concept change per
variation.
Carefully check
1 variation answers 1 question
E.g.
Survey's show Low Price is
very important to shoppers,
but this key info isn't shown
clearly at the Product Page.
E.g.
Does showing the £ or % saving
(at product page)
boost shopping KPIs?
Which is best?

LET’S TAKE A LOOK AT
AN EXAMPLE 👀
A REAL TEST

PROBLEM - BACKED BY CUSTOMER DATA
P
Real Example: Test idea using data, designed to answer a question. This guides us to test and learn.
Perhaps we can boost KPIs by showing the value / discount.
Original
Surveys revealed that Low Price is very important,
but at the Product Page the saving VALUE AMOUNT
isn't currently shown.

Original
Q: Will we get a boost to shopping metrics

if we show the saving in £ or %, at the Product Page?

Which is best?
QUESTION(S)
Q

S SOLUTION IDEAS - W/ INPUT FROM WIDER TEAM
Original V1 V2
Q: Will we get a boost to shopping metrics

if we show the saving in £ or %, at the Product Page?

Which is best?

KPI: Conversion Transaction Rate

Secondary Metrics: Revenue per Session, Pages per Session, Add to Carts
HOW TO MEASURE - PRIMARY KPI, SECONDARY KPIS
M

SURPRISING RESULTS - NO WINNER - BUT WE LEARNED

SURPRISING RESULTS - NO WINNER - BUT WE LEARNED
-5.7% CVR (LOSER)
-1.43% CVR (FLAT)
Original V1 V2
Original had 97% chance to be best

Original V1
We tried a new solution…

New Q: If we simply show the item is on sale, will we get a lift in key Shopping KPIs?
NEXT ITERATION

We tried a new solution…

New Q: If we simply show the item is on sale, will we get a lift in key Shopping KPIs?
NEXT ITERATION
Original V1
+7% KPI (WINNER)
V1 had 96% chance to beat original

Make small bets. Learn. Iterate or move on.
PQ FRAMEWORK
ENSURES YOU WILL
TEST TO LEARN.
AND AVOID
"TEST TO WIN"
You will win more often than testing to win.
Note: After gaining some learnings,
you can do some 'test to win' tests → but strive to still learn!

TEST TO LEARN.
WHY DOES THIS WORK?
Surveys revealed that our customers love low prices.
But surprisingly - showing the saving value or percent
discount doesn't resonate with shoppers (at the PDP).
Instead, they just want to know the item is "on sale".
Brainstorm new ideas: We should try optimizing experiences
and solutions, around shopping for sale items…
e.g. Highlight "on sale" at various stages of the user journey
You will gain strategic business knowledge, like…

IN SUMMARY
6 PRACTICAL TIPS
TO IMPROVE YOUR CRO PROGRAM
04.

6 PRACTICAL TIPS
TO IMPROVE YOUR CRO PROGRAM
TRY THE PQ FRAMEWORK
LOG TEST LEARNINGS
IN SUMMARY…
TEST TO LEARN AND NEVER LOSE
MOVE ON OR ITERATE?
03.
04.
05.
06.
DO THE HEAVY LIFTING
01.
AVOID CRO PROGRAM KILLERS
02.
BONUS:
QA Tests on real devices
Don't trust surprising data! (Twyman's Law)

VWO Webinar 2022
Talk by Chris Marsh
Chris Marsh
Freelance CRO Specialist

DashofCX.com
twitter @ChrisMarsh_uk
THANKS! ONTO Q&A
TRY THE PQ FRAMEWORK TO TEST & LEARN
LOG TEST LEARNINGS
TEST TO LEARN AND NEVER LOSE
MOVE ON OR ITERATE?
03.
04.
05.
06.
DO THE HEAVY LIFTING
01.
AVOID CRO PROGRAM KILLERS
02.
IN SUMMARY…
Also, let's connect on LinkedIn…
LINKEDIN.COM/IN/DATACHRIS
BONUS:
QA Tests on real devices
Don't trust surprising data! (Twyman's Law)
I'm always happy to chat about CRO and help

“I Added +$3M to Client Revenue With CRO & UX”

Recommended

Recommended

More Related Content

More from VWO

More from VWO (20)

“I Added +$3M to Client Revenue With CRO & UX”