CRO/CXO, Experiments and A/B testing have gained a lot of traction in the last years, hopefully also within your company; When optimizing, a lot of commerce teams seem stuck experimenting with minor frontend changes like tekst, images or layout. While still valuable to do this, you could benefit much more from the experimentation mindset when applying this to not just your frontend, but to across all aspects of your business Guido shows you how using Packaged Business Capabilities can give your commerce team the freedom to do just that.

1. Still A/B testing button colors?
BIGGER
Think

2. 3 Questions

3. Who works at a
company/agency
that is A/B testing?
Q1

4. Who is involved in
the A/B testing
process themselves?
Q2

5. Who runs 10+ A/B
tests a month?
Q3

6. To give you some context…
Guido X Jansen
Global Business & Technology Evangelist @ Spryker
● Applied Cognitive Psychologist
● Working in E-com & Conversion Optimization (CRO) since 2008
● Ran 1000+ experiments
● Build several international CRO teams at Ecom merchants
● Won the individual Experimentation Culture Award in 2020
● Host of the CRO.CAFE Podcast
gui.do/linkedin
@guido

7. I’m fascinated by the brain
and human behaviour

8. Why think BIGGER?

9. I made a lot of mistakes…

10. I liked the details a bit too much…

11. I was a control freak

12. I got too scientific about it

13. I forgot about the stakeholders

14. And I feel I’m not the only one…

15. So let’s talk about Small vs Big…

16. Sidenote: Framing your Experiments

17. Everyone is already experimenting (unwittingly)

18. CRO is about validation and raising the quality

19. Part 1: Small versus Big

20. I might get in trouble for this…

21. Small vs Big: Validation types
Classic A/B test
Small = single change
Classic Split-URL test
Big = Multiple changes all at once
Classic Multivariate testing
(MVT)

22. Small vs Big: Impact
source:
.org
+4.3%
Median Impact from
376 “Small” experiments
+15%
Median Impact from
43 “Big” experiments
Database of 419 Experiments

23. Small vs Big: Required Sample Size
4.3
162,289
15
13,533
Small test
Big test
source:
EvanMiller.org

24. Small vs Big: Test Duration
Duration of 3 small A/B tests (4.3% compounded to 13.5%)
Duration of 1 big A/B test (15%)
973.734 sample size
27.066 sample size

25. Running BIG experiments…
Decrease
Time
Increase
Impact

26. Perfect for smaller companies
Lower
Traffic
Higher
Upside

27. TIME IS THE MOST
VALUABLE
RESOURCE
CLEAR FOCUS IS
MORE NEEDED
THAN EVER

28. Part 2: Big

29. Big Example: Amazon product page redesign
Full case: https://goodui.org/leaks/amazons-beautifully-designed-and-failed-three-vs-two-column-layout-experiment/
Version B
Rejected

30. Big Example: Netflix homepage redesign
Full case: https://goodui.org/leaks/netflix-rejected-all-these-tested-homepage-variations-perhaps-this-one-will-help/
Version B
Rejected

31. Mixing the odds isn’t great…
Mixed
Probabilities
✘
+
+
+ -
-
?
?
Positive
Probabilities
✔
+
+
+ +

32. Increase your probability for success
● Combine small and big tests
○ Run small experiments for Cause-Effect
○ Run big experiments for big impact
● Apply patterns that worked before
○ GoodUI.org Leaks & Patterns
○ Extrapolate your own successful patterns
○ Properly document your own experiments

33. Example: Thomasnet.com Signup Page UI Changes
Full case: https://goodui.org/blog/6-signup-form-ui-changes-with-17-more-signups-for-thomasnet/
Test result details
+17%
Signups
source:
.org

34. Example: KensingtonTours.com 12 landingpage changes
Full case: https://goodui.org/blog/12-landing-page-patterns-for-42-more-leads-on-kensingtontours.com/
Test result details
+42%
Leads
source:
.org

35. Increase your probability for success
For Cause-Effect
Run Small Experiments
For Biggest Impact
Run Big Experiments

36. Build pattern prioritization feedback loops
Prioritize based on and adjust weights based on your own test outcomes.
Device
Page type
Metric
Tag
…
Template available > DM me!

37. Part 3:

38. Don’t stop at the frontend!

39. Business Capabilities & Features to A/B test
Cross-sell &
Upsell
Catalog
Management
CMS
Internationalization
Discounts &
Promotions
Cart
Gift Cards
Checkout CRM
Company
Account
Back Office
Forms
Order Management
Search & Filter
CMS
SEO
Payment
Personalization
Mailing &
Communication
Media
Management
Navigation
Multi-
Channel
Technology Partner Integrations Wish List
Workflow & Process
Management
Product
Management
Payments
Tax
Shipment Ratings & Reviews
Shopping List
Frontend
Frontend
Price

40. How enables you to do this
Spryker Cloud Commerce OS is a B2B,
B2C and marketplace solution renowned
for its ease of use, flexibility, and speed.
Fully modular system
● 120+ Features
● 30+ Business Capabilities
Spryker.com

41. Spryker solution partners that can help you run BIGGER experiments
40+ Solution
Partners @
Spryker.com

42. 14K+ Subscribers, 250+ Episodes on tap
English | Spanish | Dutch | German

43. So can we NEVER test button colors…?

46. Thanks for listening!
Guido X Jansen
Global Business & Technology Evangelist @ Spryker
gui.do/linkedin
Find me and my team at the booth!

47. End

49. Templates

50. Intro
"Stop running minor frontend experiments: experiment with ALL your business capabilities!"
or "Still A/B testing your buttons? You need to think (much) bigger..."
● CRO/CXO, Experiments and A/B testing have gained a lot of traction in the last years, hopefully also within your company;
● When optimizing, a lot of commerce teams seem stuck experimenting with minor frontend changes like tekst, images or layout.
● While still valuable to do this, you could benefit much more from the experimentation mindset when applying this to not just your
frontend, but to across all aspects of your business
● Guido shows you how using Packaged Business Capabilities can give your commerce team the freedom to do just that.

51. Title
● …

52. Real-life A/B test example
source:
.org