The right technical support for your A/B testing program requires so much more than just finding someone who knows HTML, CSS, and JS. You will only truly achieve the benefits of experimentation when your development is both efficient and sits correctly within a wider process. Tune in to listen to Jonny Longden, Director of Digital Experience at Journey Further, to find out how to get the right development support in place and ensure you are getting the most from your A/B testing program. Key Take Aways Getting it wrong is a huge drain on your time as a CRO. Not all developers are the same. A/B test development is a rare skill The process that unifies A/B testing with production development is where efficient innovation happens

1. 1

2. ©2022 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
Jonny Longden
Over the last 13 years I have helped the likes of Sky,
Visa, Manchester United, Nike, O2, Mvideo, Principal
Hotels and Nokia put analytics and experimentation at
the heart of their digital innovation and engineering.
Digital Experience Director

3. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
3
1. Types of front-end development in A/B testing
2. How A/B testing delivers value
3. Where it goes wrong
4. How to get it right
Agenda
3

4. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
Types of front-end
development for
A/B testing
4

5. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
5
WYSIWYG
Beg, borrow,
steal
Allocated
support from
production dev
agency/team
Dedicated
developer(s)
Product-team
(centralised)
Product-team
(CoE)
The evolution of A/B test development

6. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
It doesn’t matter how ‘advanced’ a
business is, they can still be getting
it very wrong.
6

7. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
How A/B testing
delivers value
7

8. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
8
Rev.
Impact
Idea 1 Idea 2 Idea 3 Idea 4 Idea 5 Idea 6 Idea 7 Idea 8 Idea 9 Idea 10
+£150k £0 £0 -£90k “£0 £0 £0 +£60k £0 -£25k
Dev £6k £1.5k £0.75k £5k £12k £3k £0.75k £4k £0.4k £2.5k
£95k net
revenue
impact
£35.9k dev
cost
2.65 ROI
The financial value of experimentation

9. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
9
Rev.
Impact
Idea 1 Idea 2 Idea 3 Idea 4 Idea 5 Idea 6 Idea 7 Idea 8 Idea 9 Idea 10
+£150k £0 £0 -£90k “£0 £0 £0 +£60k £0 -£25k
Dev £6k £1.5k £0.75k £5k £12k £3k £0.75k £4k £0.4k £2.5k
Exp.
Win No diff No diff Lose. No diff No diff No diff Win No diff Lose
£1.2k £0.3k £0.15k £1k £2.4k £0.6k £0.15k £0.8k £0k £0.5k
£95k £210k
net revenue
impact
£35.9k £15.6k
dev cost
2.65 13.5 ROI
The financial value of experimentation

10. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
10
De-risked web development (examples)
Should we implement PayPal/Klarna Painted door test
Should be allow people to review our
products on our website
Offer basic function to collect a review
to see how many people click it
Should we invest in new product
photography?
Take new shots of a few key products
and test

11. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
To get the greatest
business benefits of
A/B Testing, your
development needs to
be efficient.
11

12. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
Where it goes
wrong: the five
big errors.
12

13. 13
©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
It takes so long for a CRO to brief and
manage a developer, they may as well do it
themself.
➔ Arduous briefing processes
➔ Developer does not see the overall
objective
➔ No design initiative; needing
exact instructions
Micro-management of
developers

14. 14
©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
Sourcing anyone who knows HTML,
CSS and JS without considering the
skills required to build inside A/B tools.
➔ Adapting existing code and
ensuring ‘graceful failure’ is
different to writing new code
➔ Code behaves differently in A/B
testing tools e.g. SPAs need to
be worked around
➔ Browser-centric not code-
centric
Thinking the skills
required are just ‘coding’

15. 15
©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
Developers follow all the same processes
as for a full production feature release.
➔ Too much browser testing (can just
exclude all old browsers in a test)
➔ Not as much cross-feature testing
required (also much easier to back
out a test)
➔ Approvals and QA levels not
always required when the test
is temporary
Copying ‘production’
processes

16. 16
©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
Using developers who would rather be doing
more complex coding projects.
➔ Deprioritization
➔ High turnover, broken continuity
➔ Slow progress
Unfulfilled developers
stalling progress

17. 17
©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
Experimentation is situated incorrectly in the
process.
➔ Full feature builds with release
testing
➔ Building for permanence
➔ No deconstruction of ideas
Failing to understand
efficient validation

18. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
How to get it right
18

19. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
19
Developer skill sets - the right developer(s)
Technical
Operations
Creative
Initiative
Developer
Technical
● Deep knowledge of the
tool(s)
● Nuance of coding for
experimentation
● Expert in driving
efficiency
● Broader understanding of
data, analytics etc.
Initiative
● Bought into the wider
process of
experimentation
● Participating in ideation
and creation
● Ownership of the wider
aspects of the test
Operations
● Able to manage efficient
QA processes as
appropriate
● Able to fit into validation
processes and suggest
efficient testing
Creative
● Design and UX flair as
well as coding skills
● Wider appreciation of
processes
● Creative application of
efficiencies

20. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
20
Feature
MVP
Validation/
POC
De-risked development processes

21. 01
02
03
04
©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
21
Use a specialist
agency
Train/involve your
developers in CRO
Rotate developers
(product teams / in-
house engineering)
Have a separate
backlog and dev
queue for
supporting
experimentation
21

22. ©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
©2023 Journey
Further. All rights
reserved. All related
logos, assets
messaging are
trademarks of
Journey Further Ltd
22
Q&A

23. Thank
you
23
23