Presentation I gave at Measurefest London on Nov 9th 2017.

1. © Randstad Group Communications, July 2017 | 1
preparing for AI:
our journey to new
optimization methods

2. how can we make
the next step?

3. how can we
maximize output
(in terms of learnings and conversion uplift)
with fixed resources?

4. faster optimization loops!

5. CRO is tough!
© Randstad

6. optimizing for the “average user”
does (s)he exist?
© Randstad

7. I need more resources!!!
get in line
© Randstad

8. too many variables impacting conversion!
● Customer persona’s
● Countries
● Budgets
● Customer history
● Regulations
● Colors
● Flower seasons
● Product types
● Holiday peaks
● Time of day/week
● Purchasing power
● Culture
● Devices
● Social media platforms
● Acquisition channels
● Symbols
● Step in Customer Journey
● Tech adoption level
● Accessoires
● Multi-channel funnels
● Etiquette
● Consumer vs Corporate
● Platform
● Different payment systems
● Brand awareness/perception
● Logistic situation
● Aesthetics
● Traditions
● Social context
● Weather
● Competition campaigns
● Consumer state-of-mind
● Competitor A/B test
● ...

9. and to make it worse…
© Randstad

10. these variables keep changing
all the ffing time!
© Randstad

12. but first
some context
© Randstad

13. Guido Jansen
global CRO Lead
randstad

14. ||
randstad in numbers (year-end 2016)
second HR services provider worldwide.
€ 20.7
billion in revenue.
32,280
avg. corporate employees.
626,300
people employed every day.
183,900
number of permanent placements.
4,752
offices in 39 countries.
45.2
% women in leadership positions.
© Randstad 2017 14

15. Previously:
Chief Psychology Officer / CRO Manager

16. ||
CRO
building blocks.
© Randstad 2017
ExperimentsPsychologyQual/Quant Data

17. ||
our global presence.
© Randstad 2017
Andorra Luxembourg
Argentina Malaysia
Australia Mexico
Austria Monaco
Belgium Netherlands
Brazil New Zealand
Canada Norway
Chile Poland
China Portugal
Czech Republic Romania
Denmark Sweden
France Singapore
Germany Spain
Greece Switzerland
Hong Kong Turkey
Hungary United Arab Emirates
India United Kingdom
Ireland United States
Italy Uruguay
Japan
(represents over 90% of the global HR services
market.)

18. ||
our brands.
© Randstad 2017

19. Live Testing
Complete/Analysis
Deployments
Design
Development
Q&A
Pending approval
Approved
Idea
Hypotheses
Full testing plan
our optimization loop

22. slight warning…
...but slides will be shared!
there will be
data & details

23. how can we make
the next step?

24. How are others doing this?

25. “[Booking.com’s] utilization of A/B testing …
drives higher conversion across it’s entire
platform, resulting in conversion levels 2-3x
the industry average”
Source: Evercore Equity Research

26. Sadly, when you test more,
the % of tests with an uplift will go down.

27. When A/B testing a fully optimized site,
the % of tests with an uplift is low(er).

28. 1 in 5 tests with uplift
1 in 10 tests with uplift
Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017

29. How to get more tests with an uplift?

30. 10-20 concurrent test
1000 concurrent tests
Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017

31. 1 optimization team
75 product teams
Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017

32. Of course, we are not ...

33. ...but we do know a thing (or 2) about
experiments...

34. ...so let’s experiment with
the way we experiment!

35. Let’s look for solutions...
© Randstad
35

36. (evolutionary) AI/ML?
sounds cool, tell me more!
© Randstad

37. Possible applications
© Randstad
37

38. Different applications of AI in E-commerce
● Catalog optimization ← already tried this
● Personalization
● RTB / Advertising
● Chatbots
● Virtual personal shopper
● Improved optimization process
● ...

39. We tried AI catalog optimization

40. Previous experience:
● “Black box”
● No (perceived) control for E-commerce managers
● No proven uplift in CR
● Hard to do for (emotional) consumers buying for someone
else?

41. but still… the AI promise is so tantalizing...
MORE CAPABLE
• Test your entire funnel in a
single experiment
ADAPTIVE
• Evolves with your audience
• Always-on optimization
GREATER IMPACT
• More tests = better results
• Focus on marketing, not test
administration
FASTER AND
MORE EFFICIENT
• Tests dozens of ideas
• Test 1,000,000s of
combinations
• Get to results faster
TIME-SAVING
• Fully automated
• Eliminates tedious test
administration
• Handles the data science

42. © Randstad
42
AI-POWERED CONVERSION OPTIMIZATION

43. “Fractional Factorial MVT + Evolution”

44. MVT...

45. … + evolution

46. How the “evolution” part works...
+ 0 - + - + 0 + +

47. How the “evolution” part works...
+ 0 - + - + 0 + +

48. How the “evolution” part works...

49. How the “evolution” part works...

50. How the “evolution” part works...

51. How the “evolution” part works...

52. Evolutionary Algorithms

53. Evolutionary Algorithms

54. Evolutionary Algorithms

55. Evolutionary Algorithms

56. Evolutionary Algorithms

57. Evolutionary Algorithms

58. Evolutionary Algorithms
+ 8.7%
PERFORMANCE OF BEST-
PERFORMING CANDIDATE
(VS. CONTROL)
+4.5%
+5.3%
+6%
…
…

59. Multipage Funnel Optimization
Home PDP Cart

60. Sentient Ascend
Tooling:
Implementation partner:

61. $143M
in total funding
9 YEARS
platform &
technology
development
35 PATENTS
submitted to date; 11 patents
issued
100+
EMPLOYEES
HQ in San Francisco with offices in San
Jose and Hong Kong
Founded by the team that developed
the core technology behind Siri

62. Sentient Ascend basic principles
DEEP
LEARNING BAYESIAN
EVOLUTIONARY
ALGORITHMS NEUROEVOLUTION

63. Tested in multiple industries
FINANCE
HEALTH
CARE
INSURANCE
CYBERSECURITY
AGRICULTURE

64. BUT...
does it work for E-commerce…?
© Randstad

65. Current brands experimenting with Sentient.ai ßeta

66. Sentient E-
commerce ßeta
© Randstad

67. Countries selected for trial
The Netherlands Norway Sweden

68. 8 elements on PDP, 2 variants each
● Header - less content
● Image position - left to right
● Add progress bar - feed forward
● Social sentence (under CTA, Hobson’s choice +1)
● Product info - full width / less prominent
● Price partitioning
● USP Bar icons/text - different content
● USP Bar position - to the top

69. ||
our main questions
• Does it work?
• How much faster is it?
• Is it worth it?
© Randstad 2017

70. (1st) baseline active: end of Feb
Live: March 20th
End: June 3rd

71. NL
3 generations, 23 variants
Sample size
Full factorial MVT requires 1M users
Sentient tested 125K users (12,5%)

72. NL - Time Series

73. NL - Best 20 variants
Best Improvement over control: 2.5%-point (12.7%)

74. Significant?
Nope...
Too many variants/ Not enough visitors...

75. NO
4 generations, 28 variants
Sample size
Full factorial MVT requires 444K users
Sentient tested 75K users (16,8%)

76. NO - Time Series

77. NO - Best 20 variants
Best Improvement over control: 4.6%-point (12.6%)

78. Significant?
Nope...
Too many variants/ Not enough visitors...

79. SE
4 generations, 28 variants
Sample size
Full factorial MVT requires 1.2M users
Sentient tested 351K users (29,3%)

80. SE - Time Series

81. SE - Best 20 variants
Best Improvement over control: 2.8%-point (6.9%)

82. Significant?
Yes!
@ CI 95%, not @ CI 99%

83. Best variants
NL
Variant 8, generation 1
● USP Bar Icons/Text
SE
Variant 26, generation 4
● USP Bar Icons/Text
● Price perception
● Product info
● Call to action
● Progress Bar
NO
Variant 8, generation 1
● USP Bar Icons/Text

84. Control Winning Variation

85. But can we trust the
numbers from
Sentient?
© Randstad

86. A/B test the winner to validate
Significant (frequentist; 90% confidence; 1-sided)
● Add-to-cart uplift: 3.9% (p-value 0.0161, Power: 95,9%)
● Conversion rate uplift: 5,3% (p-value 0.0361, Power 89,6%)
Not significant
● Average Order Value

87. How much does it add to our
bottom line?

88. Business case:
● Costs: 165K (subscription + support + salary)
● Expected uplift: 500K-1M profit
● Minimal CR uplift needed for break even: 1.25%

89. Final Thoughts
© Randstad

90. ● Steered on CR, not on RPU
● GA data issues
● Sentient dashboard UX issues
● Calculator to determine ideal values and estimated runtime
would be very helpful
Specific Sentient caveats/ needed improvements to ßeta

91. Pro/Cons: Regular A/B/MvT testing
Pro:
● More control
● Stricter hypothesis testing
● ...
Con:
● Slow
● Less variants
● Needs a lot of traffic
● Limited test capacity
● ...

92. Pro/Cons: Testing with AI
Pro:
● “Time per tested variant” much
shorter
● End up with “uncommon”
variation
● Requires less traffic (around
75% less)
● Higher ROI
● ...
Con:
● More extensive setup process
● Black box (?)
● Don’t know exactly how fast
generations will follow...
● Garbage in, Garbage out
● Completely freaks out our local
Ecom managers.
● ...

93. “I just followed instructions”
‘Compu’er says no’

94. ||
our main questions
• Does it work?
• How much faster is it?
• Is it worth it?
© Randstad 2017

95. Did Sentient deliver on their promise...?
MORE CAPABLE
• Test your entire funnel in a
single experiment
ADAPTIVE
• Evolves with your audience
• Always-on optimization
GREATER IMPACT
• More tests = better results
• Focus on marketing, not test
administration
FASTER AND
MORE EFFICIENT
• Tests dozens of ideas
• Test 1,000,000s of
combinations
• Get to results faster
TIME-SAVING
• Fully automated
• Eliminates tedious test
administration
• Handles the data science

96. © Randstad |
will this make the CRO team
obsolete?
© Randstad

97. |
well...
..I’m hiring :)
© Randstad

98. Link to all slides:
twitter.com/guido
© Randstad