AB Testing and UX - a love story with numbers and people Slides from the NUX6 talk by Craig Sullivan, Friday 27th October 2017. 2017.nuxconf.uk / nuxuk.org Synopsis: What’s wrong with the web these days? The mobile experience sucks. The customer experience sucks. It doesn’t work. It’s too hard to use. The text is too small. Nobody measures this happening. The interaction patterns suck. Nobody ever calls up to complain but nobody does anything anyway. Millions of people lose countless days to friction, poor design and frustrating moments on their devices. There may be thousands of things you can fix that look promising – but how do you know where to start? What if you could measure what sucked, where it sucked and how big the problem was? Using lightweight research methods and tools, you can stop making excuses and start knowing exactly what to do. Life becomes much simpler and easier with a scientific method of optimising growth or delight within your product. Craig has trained over 500 people on how to measure and optimise their product experience, finding 100M of ‘lost revenue’ using just one of the techniques you will learn. With reports, checklists, downloadable templates and toolkits for every budget and stage of growth – you can stop guessing tomorrow.

1. CRAIG SULLIVAN
Senior Optimisation
Consultant
@OptimiseOrDie
24TH OCTOBER 2013
EMETRICS LONDON
A LOVE STORY WITH
NUMBERS AND PEOPLE
27T H OCT 2017
NUX MANCHESTER
Optimiser Of
Everything
AB TESTING AND UX

2. Digital Product Design is
And we’re all going to die…
@OptimiseOrDie
bit.ly/ProductPoo

3. BACK TO 1999
@OptimiseOrDie
• Now I’m known as ‘that testing guy’
• But I started life as a UX practitioner
• >600 hours moderating in 3 years
• Everything I did started here
• Where did it take me?

4. • UX, Analytics, Tools, blended.
• I coach people and teams on
how to experiment profitably
• I’m a plumber looking for
‘ leaks’ in the delight or revenue
for a business
• Most companies don’t know
• This is my job
@OptimiseOrDie
TODAY

5. Why do I get so angry?
Because We are
F***ing Stupid!
@OptimiseOrDie

6. @OptimiseOrDieAnnual Sales in DE/FR/UK/IT/ES = £4.5Bn (2015)
After Millions of Years of Evolution…

7. @OptimiseOrDie

8. @OptimiseOrDie
3 minutes later…

9. Reddit Explains…
“If it comes off in one, it's gonna be
banging! If you’re picking plastic out
of your dinner for ten minutes, it's
probably gonna taste like shit.”
@OptimiseOrDie

10. Quick Calculation:
• £4.5Bn Sales @ £3.5 average price
• Assume 60% are failures
• Costs 90 seconds of wasted time
• 72514285714 seconds wasted
• 1208571428 minutes
• 20142857 hours
• 839,285 days of life - wasted on friction!
• What about injuries? Food wastage? Ennui?
@OptimiseOrDie

11. @OptimiseOrDie
How hard is this to fix?

12. A Packaging Manifesto:
1. Packaging should never require a tool or knife to open
2. Packaging should not require finger control or strength
which discriminates against population groups
3. Any packaging must be capable of being returned to its
original state to preserve any unused contents
@OptimiseOrDie

13. SECTION TITLE
SWINGING IN SCOTLAND
@OptimiseOrDie

14. SECTION TITLE
“AB TESTING FACE”

15. “Burning Rubber Doughnuts in the
Car Park of User Experience ”
“Testing Debt, man”

16. @OptimiseOrDie
1. Change Growth TrajectoryWould you do it in retail?

17. Awful terrible, unbelievably bad shit 80%
Passable 15%
OK 4%
<1%
5 YEARS AND 130 COMPANIES
Good

18. @OptimiseOrDie
“We are a data driven and
customer centric organisation…”
(but none of us have watched,
read or listened to customer
feedback in a year)

19. @OptimiseOrDie
“We have Session Replay,
Heatmaps, Form Analytics and
Voice of Customer Feedback”
(we just don’t look or act
on what they tell us)

20. @OptimiseOrDie
“We have agile product
development teams”
(but we haven’t reformed
anywhere else - signoff
takes a month)

21. @OptimiseOrDie
“We have a huge Conversion
Optimisation Team of 6 people”
(but they’re working
in a Silo, tinkering with button
testing or trivial insights)

22. @OptimiseOrDie
“We apply lean thinking at all
levels of the organisation”
(but we haven’t
changed the way
teams and managers
actually work)

23. @OptimiseOrDie
“We have the best tools,
methodologies and team around”
(but absolutely nothing
useful happens)

24. @OptimiseOrDie
Current State of the Internets

25. @OptimiseOrDie
Is this a “Landing Page Experience”?

26. www.ba.com/gatwick

27. www.ba.com/gatwick

28. <2-3mm
Last night I saw an advert…

29. Last night I saw an advert…

30. Are you f***ing kidding me?

31. And it gets worse!

32. And it gets worse!

33. Screwing up Single Fields!
Spell Checker Enabled Wrong Keyboard Chosen
@LukeW

35. The Pinnacle of Optimisation?

36. Point Optimised Persuasion Ninjas

37. The Ikea of Psychological Ninja Messaging?
• Like a ‘Franchise’ store
• Every space competing directly
• Maxed out on persuasion
• Point optimised
• Works ‘on average’
• Holistic Experience?
• How do I actually FEEL?

38. The Ikea of Psychological Ninja Messaging?
CMA to probe Hotels: bbc.in/2hfh70r

39. @OptimiseOrDie
Persuasion Motivation Usability Broken shit
How to Shi8 the Needle 101

40. @OptimiseOrDie
• Clothing 1M per month, 2 bugs
• Travel 2M per month, 13 bugs
• Telecoms 1.4M per month, 3 bugs
• Travel 0.6M per month, 1 bug (iPad)
• Event Site 2.5M per month, 11 bugs
– These were only the first bugs we found
– < 5 days time to fix – an ROI of 91,150%
– Larger than the annual IT budget for some!
How much do product defects cost?

41. When’s the last @me you checked these?
@OptimiseOrDie
Marke-ng
Whiffle
Can people read it?
Can people use it?
Does it load and work quickly?
Does it work on customer devices?

42. @OptimiseOrDie
Just add AB tes@ng for instant sparkle!

43. Why is this stuff important?
“Because an experience
is only as good as its
shittiest part.”
@OptimiseOrDie

44. • We’re swamped with data
• The underlying data is often flawed or unreliable
• We don’t know what’s wrong with our products
• We change stuff without evidence or observation
• We build stuff nobody wants or needs
• Product changes are not measured or tested
• Nobody takes stuff away from the website
• We build what internal people want, not customers
• We don’t remove costly defects and wastage
• Lean & Agile have left us with new types of debt
• Ego is the enemy of all good work
• Much Internet design is infused and driven by Ego
@OptimiseOrDie
What is going wrong?

45. Thx @sdelbecqui @ericries @ycaseau

46. @OptimiseOrDie
Lean Organisations:
Pinterest
Twitter
Linkedin
Dropbox
BitTorrent
Airbnb
FT
ING
Spotify
Booking.com

47. @OptimiseOrDie
• Copywriter
• Data scien-st
• Product Lead
• UX Researcher
• Developer
• Designer
Booking.com - Corporate Models

48. @OptimiseOrDie
FT.COM - Decoupling

49. @OptimiseOrDie
• MVP approach
• Launch as alpha, beta, pilot,
phased rollout
• This approach includes
split testing to tune,
optimize or decide!
• 100s of releases a day!
• Like getting in a shower
• Read more at labs.ft.com
FT.COM – Release Methodology

50. @OptimiseOrDie
• Small teams with direct access to publish
• Ability to set and get metrics data directly
• Tools, Autonomy, Lack of interference
• No Project Managers or Business Analysts
• Stakeholders decoupled from work
FT.COM - Corporate Models bit.ly/2mZAxH7

51. @OptimiseOrDie
• Business wants: “15% increase in 12 months”
• Business defines ‘outcomes’ – teams deliver
• No long signoff chain
• No pesky meddling during execution
• Less time wasting meetings with morons!
• Outcome not feature driven optimization
FT.COM - Decoupling bit.ly/2mZAxH7

52. @OptimiseOrDie
• Most product changes we make are a waste of time!
• The Financial Times – 50-70% of product developer time
wasted in a siloed organisational structure
• Minimum project time was “At least 18 months” for anything
• Projects were always late, underperformed, didn’t shift
metrics as expected
• Post changes, every IT project was under time & budget
• Products were iterated to hack their way to success
FT.COM – Advantages

53. @OptimiseOrDie
“Product changes and AB tests are just the
same thing to us. Where we’re moving to is
measuring and testing everything we do.
This approach tells us what works, what
doesn’t, what to keep, what to remove and
what we need to iterate.”
Financial Times – NextFT project
FT.COM – Advantages

54. • Reform the Organisation, not the tools
• Invest in analytics and people skills
• Set people free to learn new disciplines and skills
• Stop delivering ‘features’ and start doing ‘outcomes’
• Validate everything - changes, tests, personalisation
• Test everything that *can* be tested
• Stop Ego from driving product – use insights & data
• Continuous improvement rather than large releases
• Use multiple product ‘horizons’ for strategy
• Blend Lean, Agile, Data, UX and AB testing
• Let’s give you some tips to take away and use!
@OptimiseOrDie
What can we do about it?

55. @OptimiseOrDie
Conversion Research in One Hour:
bit.ly/batmantesting
Session Replay Tools for Research:
bit.ly/decibelWAW & bit.ly/ReplayTools
The List of UX Tools to Rule Them All:
bit.ly/UxTools & bit.ly/uxresourcepack
Lean Research – Further Reading:

56. @OptimiseOrDie
Lean Organisations – resources:
How Booking.com tests & organises teams:
http://bit.ly/SFrisby
Growth Team Models & Explanations:
http://bit.ly/GHmodel
ING video on Lean Corporate:
http://bit.ly/ingvideo

57. 1. Find out your Testing Bandwidth - Can you run any tests at all?
2. Stop copying stuff blindly - Test or validate changes, don’t assume
3. Get with Statistics - Doing testing? You need stats skillz
4. Run a test Calculator - How long it takes and when to stop
5. Prioritise All the Testing - Do the right tests in the right order
6. Get a Decent Hypothesis - Forget this, forget your AB testing
7. Record the device experience - Store the screen & recordings
8. Simplify your Test Decisions - Stop overcomplicating test analysis
9. Avoid Toxic AB Testing - Defects ruin your work and decisions
10. Test Across 3 Horizons of Product - How to test ahead of yourself
@OptimiseOrDie
10 AB Testing Power-ups

58. >1200 tests a year
>1000 tests live
99% chance you’re in 10 tests
7000 tests in 2011
100,000 tests last year
@BartS
1: Find Out your Testing Bandwidth

59. 1: Do you have testing bandwidth?
• This is a little Amoeba
• It hasn’t evolved to ride a bike yet
• Do you have enough traffic?
• < 500 ‘conversions’ a month?
• Work on product/market fit!
• Work that user research!
@OptimiseOrDie

60. 2: Stop Copying Stuff Blindly…
• Your customers are not the same
• Your site is not the same
• Your platform is not the same
• Your advertising isn’t the same
• The traffic isn’t the same
• The UX is not the same
• Your Device Mix is not the same
• Your mistakes are different
bit.ly/BestPUnicorns
bit.ly/BPSuccess
bit.ly/BestpFAIL
@OptimiseOrDie

61. @OptimiseOrDie
“STOP copying your competitors.
They may not know what the f***
they are doing either”
Peep Laja, ConversionXL

62. @OptimiseOrDie
“Why every Internet Marketer should be a Statistician”
http://bit.ly/1wMfs1G
“Statistical Significance does not equal Validity”
http://bit.ly/1wMfmY2
“Understanding the Cycles in your site”
http://mklnd.com/1pGSOUP
“Statistics for Dummies”
http://bit.ly/stats4dummies
3: Get With Statistics

63. @OptimiseOrDie
• Run a test calculator first
• If the answer is 9 Million years, do something else instead!
Online Dialogue Suite: bit.ly/CaLC1
VWO : bit.ly/CaLC2
Optimizely : bit.ly/CaLC3
Evan Miller : bit.ly/CaLC5
• Testing is trading time and effort vs. potential reward
4: Run a Test Calculator

64. @OptimiseOrDie
§ Quadrants
§ PIE
§ ICE
§ ICE redux
§ TIR
§ Money Model
§ Hotwire
§ PXL
5: Prioritise Everything!

65. @OptimiseOrDie
Target Users
per
week
CR % AOV Each 5% 5% li8
detected
sample
Min
Weeks
10% li8
detected
sample
Min
Weeks
15% li8
detected
sample
Min
weeks
Basket 100,000 5% £140 £35,000 48098 1 12301 1 5590 1
Landing 1 60,000 3% £190 £17,100 25566 3 6531 1 9528 1
Landing 2 18,000 0.6% £100 £540 419876 47 107513 12 48913 6
• How long will it take to see a difference of 5, 10 and 15%?
• How much sample are you going to need to ‘detect’ an effect you’re hoping for
• S-ck to tests that have decent sta-s-cal power, to save was-ng -me!
• Read these two ar-cles:
https://inbound.org/discuss/conversionxl-live-notes-slides-and-q-a-
slideshare-s-now-included#paulinemarol
https://conversionxl.com/better-way-prioritize-ab-tests/
5: Statistical Power & Money Model

66. @OptimiseOrDie
bit.ly/hyp_kit
6: Get a Proper Test Hypothesis

67. @OptimiseOrDie
Problem
Statement
Hypothesis
Valida-on
Rejected
Hypothesis Hypothesis
Valida-on
AB TEST
Build/Measure/Learn
Hypothesis
Valida-on
MVP
Build/Measure/Learn
6: Get a Hypothesis Hierarchy bit.ly/2mZAxH7

68. @OptimiseOrDie
• Store the design layout in Google Analytics or other package
• Use this to segment the results by the size of the design seen!
7: Record the Design people ‘see’

69. @OptimiseOrDie
bit.ly/Zoltar1
8: Simplify your Test Decisions

70. bit.ly/devicetesting
9: Avoid Toxic Testing using QA

71. Current Product
Op-misa-on
MVP, Next Product
Feature op-misa-on
Bedding in
Unknown Product
Discovery &
Explora-on
10: Test Across 3 Horizons bit.ly/3horiz

72. 1. Get an Analytics Audit - Broken tills mean broken company
2. Make a Device List - Prove you know this please
3. Test and Fix Broken shit - Customers don’t self report problems
4. Do Rapid Usability testing - Low cost and vital for great ideas
5. Add Session Recording - See broken stuff, get evidence, fix it
6. Run a Customer Survey - Regular surveys provide long term insight
7. Install Voice of Customer tools - This asking people shit will never catch on
8. Hire a Copywriter - Best money you’ll ever spend
9. Invest in Analytics - Google Analytics is NOT free
10. Add Form Analytics - Stop guessing and know what happens
11. Check your Web Performance - Performance makes £££
@OptimiseOrDie
11 Optimisation Power-ups

73. @OptimiseOrDie
1: Get an Analytics Audit, NOW!
“Check the tills
are working AND
recording correctly!”
@OptimiseOrDie
@CharlesMeaden

74. • What’s YOUR testing list?
• Anyone checked if it’s true?
• Use Google Analytics to tell you!
• Tip – do these as a minimum:
• iPhone 5/6
• iPad late model
• Samsung late model
• Desktop (Chrome, Edge, Safari)
bit.ly/UltimateXDO
@OptimiseOrDie
2: WHAT do they ACTUALLY use

75. @OptimiseOrDie
End Device Discrimination TODAY!
• If your shop doors aren’t wide enough?
• Is it your fault or the customer?

76. @OptimiseOrDie
Desktop Testing
Crossbrowsertesting.com
Browserstack.com
Mobile & Tablet
Deviceanywhere.com
Aws.amazon.com/device-farm
bit.ly/MobileDeviceLab
3: Test Real Customer Handsets – NOW!

77. Som, feedback
Concepts:
Usability Hub www.usabilityhub.com
Five second test www.fivesecondtest.com
No Lab Testing:
Usertesting www.usertesting.com
Userlytics www.userlytics.com
Userzoom www.userzoom.com
Ethnio www.ethnio.com
What Users Do www.whatusersdo.com
Loop11 www.loop11.com
77
4: Validation & Usability tools

78. Som, feedback
DESKTOP & LAPTOP
CamStudio (free) www.camstudio.org
Mediacam AV (cheap) www.netu2.co.uk
Silverback (Mac) www.silverbackapp.com
Screenflow (Mac) www.telestream.net
MOBILE
UX Recorder (iOS) www.uxrecorder.com
Skype Hugging bit.ly/tesTfm
Airplay Mirroring bit.ly/JnwtMo
DIARY STUDIES
Get rich Context bit.ly/multichannels 78
4: Guerrilla & Diary Research:

79. @OptimiseOrDie
4: Shitty User Research – Stop It!
What the f*** is going on here:
• Biased samples from recruitment & screening
• Ineffably bad or biased moderator scripts
• Focus on quantitative questions in qual research
• Biased online panels (serial testers, poor screening)
• Low cost does not mean good fidelity
• Moderation as a skill to me seems ‘less valued’

80. 1.5 Interviewing Tips
• Doing this stuff properly is hard
• Interviewing and testing tips:
“Don’t Make Me Think” amzn.to/1gIZEJn
“Rocket Surgery Made Easy” amzn.to/1e0hnUL
“Talking to Customers” bit.ly/1e0hT58
“Talking with Participants” bit.ly/1kKL3LE
“Don’t listen to Users” bit.ly/1cQpiIE
“Interviewing Tips” bit.ly/1fKqu03
“More interviewing Tips” bit.ly/1bmvGT
80

81. @OptimiseOrDie
5: Session Recording for “User Struggle”
• Helping you find out WHY!
Decibel Insight www.decibelinsight.com
Hotjar www.hotjar.com
Inspectlet www.inspectlet.com
SessionCam www.sessioncam.com
Clicktale www.clicktale.com
Mouseflow www.mouseflow.com
Ghostrec www.ghostrec.com
Yandex WebVisor (free) metrica.yandex.com/promo/webvisor

82. @OptimiseOrDie
SurveyGizmo www.surveygizmo.com
Google Forms www.google.com/forms/about/
Surveymonkey www.surveymonkey.co.uk
Typeform www.typeform.com
Qualtrics www.qualtrics.com
6: Run a Customer Survey

83. @OptimiseOrDie
7: Install Voice of Customer or Polling Tools
HotJar www.hotjar.com
Kampyle www.kampyle.com
Qualaroo www.qualaroo.com
Webengage www.webengage.com

84. @OptimiseOrDie
8: Pay for a Copywriter FFS

85. @OptimiseOrDie
8: The Economics of Copywriting
“Stop asking people to fill-in by writing
copy. Especially don’t ask developers to
do this! Hire staff or train them.
Bad copy on your site costs more money
than hiring someone. Good copy earns
tens or hundreds of times that cost..”

86. @OptimiseOrDie
9: Invest in Analytics
• Google Analytics is NOT free
• Your Car will not run without Oil
• Your Analytics will not run without Love
• Put budget into optimising the analytics!
It’s like writing a love note to your future self!
Do it today. Start writing it.

87. @OptimiseOrDie
10: Add Form Analytics
Formisimo
www.formisimo.com
Heap analytics
www.heapanalytics.com
Hotjar
www.hotjar.com
Google Tag Manager & Analytics
bit.ly/SimoGA
“This is so we can call you in
case there’s a problem
delivering your order.”

88. 11: Check your Web Performance
• Turn off Wi-Fi at all times!
• Use these performance tools:
Mobile & Desktop
Mobile Speed test
Google Pagespeed Insights:
• Google analytics TELLS you what’s slow:
• Google Statistics on Site Performance:
(< 2 seconds good, 2-3 just OK, 3+ bad)
@OptimiseOrDie

89. 1. Get an Analytics Audit - Broken tills mean broken company
2. Make a Device List - Prove you know this please
3. Test and Fix Broken shit - Customers don’t self report problems
4. Do Rapid Usability testing - Low cost and vital for great ideas
5. Add Session Recording - See broken stuff, get evidence, fix it
6. Run a Customer Survey - Regular surveys provide long term insight
7. Install Voice of Customer tools - This asking people shit will never catch on
8. Hire a Copywriter - Best money you’ll ever spend
9. Invest in Analytics - Google Analytics is NOT free
10. Add Form Analytics - Stop guessing and know what happens
11. Check your Web Performance - Performance makes £££
@OptimiseOrDie
11 Optimisation Power-ups

90. SO WHAT’S THE MESSAGE?

91. SECTION TITLE
1 MOST AB TESTING IS DOG POO
2 CROSS DEVICE EXPERIENCES SUCK
3 DEFECTS AND WASTAGE ARE HUGE
4 EONS ARE WASTED BUILDING USELESS STUFF
5 PEOPLE & CULTURE MAKE PROBLEMS, NOT TOOLS
6 DATA DRIVES NOTHING, PEOPLE ARE THE LENS
7 UX, ANALYTICS, TESTING & TOOLS = SUCCESS
8 TEAM PRODUCTIVITY & QUALITY STILL SUCK
9 REFORM THE COMPANY, NOT THE TECHNOLOGY
10 QUAL WITH QUANT TRIANGULATES & FOCUSES
SUMMARY

92. 92

93. @OptimiseOrDie
The Experience Optimisation Engine
(1) Pre Model Checklist
Groundwork and common
sense checks before any
modelling or research begins
(2) Rapid Modelling
Basic sketching & modelling to
prioritise focal areas for
research, reduce scope
(3) Research or Heuristic
Use upfront & rapid analytics
insights to prioritise focal areas
for research time
(4) Model
Post research – model the site
around priorities from research,
analytics & more
(5) Streaming
This is where the big pile of
issues gets turned into different
project backlogs
(6) Triage & Triangulate
Where you cluster themes or
issues, triangulate clues from
multiple tools and run sizing
against test targets
(7) Prioritisation
Where you bubble the most
promising test ideas to the top
of the stack bit.ly/CRO4Agencies

94. s zn+1 = zn
2 + c
Your are Optimising a Business:
@OptimiseOrDie

95. CRAIG SULLIVAN
Senior Optimisation
Consultant
@OptimiseOrDie
24TH OCTOBER 2013
EMETRICS LONDONOptimiser Of
Everything
THANK YOU!
Manchester Workshop Next Week:
bit.ly/MancClass
20 hours of training for £380:
bit.ly/CRO4Agencies
Get this Deck:
http://bit.ly/ProductPoo
Read my stuff:
medium.com/@optimiseordie
@OptimiseOrDie