“In God we trust, all others must bring data”. Intuition, experience and well known patterns may give us good indications of successful ideas and features, but nothing gets closer to the truth than data analysis and A/B testing. In this workshop, we’ll show how we do experimentation at Booking: what we test, how to get data through templates and JavaScript, and how we analyse the resulting metrics. We’ll live-code examples, see all potential caveats of dealing with the user tracking on the client-side, and show existent tools you can use to test your own ideas.
13. Data about the website is generated
as users browse through pages and
do their tasks.
14. product added to cart
number of products added
purchase finished
average price per purchase
number of products seen
user has logged in
used guest checkout
customer service calls
…
15. When there’s enough information to
make a decision, you can either stop
the test (keeping version A) or choose
version B, directing all traffic to it.
16. Buy now Buy now
Duration: 14 days
Visitors: 45.140 (22.570 per variant)
339 (1.5%) 407 (1.8%)
20% up
144.500 COP 147.390 COP
2% up
Number of purchases:
Average price:
45. • Who will participate?
• What is the primary metric?
• Any secondary impacts?
• How will it be implemented?
46. • Users from Argentina, Bolivia, Brazil, Chile,
Colombia, Ecuador, Guyana, Paraguay, Peru,
Suriname, Uruguay and Venezuela, on all
platforms
• Conversion (net bookings) uplift is expected
• We expect more returning customers
53. When you reach the expected runtime,
number of visitors or effect, look at the
data and take a decision.
54. product added to cart
number of products added
purchase finished
average price per purchase
number of products seen
user has logged in
used guest checkout
customer service calls
…
56. • How were the primary and secondary
metrics impacted?
• What were the results isolated by each
country?
• What were the results isolated by each
language?
• Did any particular platform (desktop,
mobile devices, tablets) perform better?
• Was the impact on returning customers
any higher than first time visitors?
57. Based on the gathered data,
plan for next steps.
58. • Should we add a copy to the flag?
• Should we add a tooltip to the flag?
• Should we increase/decrease the flag size?
• Should we restrict it just for desktop users?
• Should we try this for a single country, or
other countries?