Getting it Booking right

Ge#ng
it
Booking
right
–

Mobile
apps

Iaroslav,

Khramov

Team
Leader
&

Android
developer

Iaroslav Khramov
TL & Android developer

About me

Making Booking right apps
ExperimenAng

•  Why

•  How

•  What
do
you
achieve

Developing

TesAng

Releasing

Booking.com
•  Founded
and
based
in
Amsterdam

•  Biggest
accommodaAon
plaKorm
in
the
world

•  >
8,000
dedicated
employees

Mobile
apps:

•  iOS
&
Android

•  7
app
development
teams

•  ~35
apps
developers

Booking.com apps
Android

•  >
10M
downloads
*

•  raAng
4.6
out
of
5
*

iOS

•  raAng
“4+”
out
of
5
**

*
according
to
Google
Play
Store
on
23/03/2015

**
according
to
Apple
Store
on
23/03/2015

Facebook

Vkontakte

Instagram

YouTube

Gmail

Inbox

WhatsApp

4.0
/
5
*

4.2
/
5
*

4.5
/
5
*

4.1
/
5
*

4.3
/
5
*

4.2
/
5
*

4.4
/
5
*

Our challenges
•  Users
all
around
the
world

-‐  Millions
of
users

-‐  Our
user
doesn’t
have
a
unique
proﬁle

-‐  High
user
expectaAons

•  40+
languages

•  >
600k
properAes

•  expert
opinion
vs.
users
opinion

What do we want to achieve?
Product
that
is:

•  User
friendly

•  TrusKul

•  Reliable

•  …

What do we want to achieve?
Product
that
is:

•  User
friendly

•  TrusKul

•  Reliable

•  …

Provide our users the best experience
before, during and after making a
booking

Are you sure?
•  Are
you
sure
that
new
design
is
good?

•  Are
you
sure
this
change
will
not
break
the
app?

•  Are
you
sure
customers
will
like
our
new
feature?

•  Are
you
sure
….
?

Variant
A

Lek
hand
up

Variant
B

Right
hand
up

Can
you
please
stand
up

Lets play!

Which got more money on Kickstarter?
Smartphone
Controlled

Paper
Airplane

Jot:
CapaciAve
Touch
Stylus

A B

Winner: Version B
A B

Winner: Version B
A B
People with hand up - please take a
seat

Which variation performed better?
A B


Winner:
Version
A

“The
truth
is,
someAmes
it’s
bener
to
have
no
proof
than

low
proof.”


Winner:
Version
A

“The
truth
is,
someAmes
it’s
bener
to
have
no
proof
than

low
proof.”

People with hand up - please take
a seat

Winner:
Version
A

“Like
most
people,
if
you
guessed
that
Version
B
won,
you

are
wrong.
Version
A
surprisingly
had
a
24%
increase
in

submissions
without
using
an
image
on
the
page.”

Which got more requests?

Winner:
Version
A

“Like
most
people,
if
you
guessed
that
Version
B
won,
you

are
wrong.
Version
A
surprisingly
had
a
24%
increase
in

submissions
without
using
an
image
on
the
page.”

Which got more requests?
People with hand up - please
take a seat

What is better for conversion?
A B


Winner:
Version
A

In
this
example
removing
the
social
bunons
resulted
in
an

increased
conversion
to
the
basket
of
11.9%.


Winner:
Version
A

In
this
example
removing
the
social
bunons
resulted
in
an

increased
conversion
to
the
basket
of
11.9%.

a seat

More completions?

Winner:
Version
B

“…surely
adding
an
eTrust
image
would
improve
form

compleAons,
it
makes
everyone
feel
safe
and
secure,

doesn’t
it?
The
answer
is
no.
Although
it’s
only
a
slight
rise

of
12.6%,
Version
B,
without
the
image,
proved
to
be
more

eﬀecAve.”

More completions?

Winner:
Version
B

“…surely
adding
an
eTrust
image
would
improve
form

compleAons,
it
makes
everyone
feel
safe
and
secure,

doesn’t
it?
The
answer
is
no.
Although
it’s
only
a
slight
rise

of
12.6%,
Version
B,
without
the
image,
proved
to
be
more

eﬀecAve.”

People with hand up - please take a
seat

Which Copy Increased Trial Sign-Ups?
A B


Winner:
Version
A

“In
this
test,
Version
A
increased
sign-‐ups
by
38%
–
a
big

rise.
However,
your
gut
feeling
might
iniAally
be
that

version
B
is
the
bener
design.”


Winner:
Version
A

“In
this
test,
Version
A
increased
sign-‐ups
by
38%
–
a
big

rise.
However,
your
gut
feeling
might
iniAally
be
that

version
B
is
the
bener
design.”

a seat

Decision-making using data: A/B testing
•  When
should
you
should
run
an
experiment
(use
A/B

tesAng)?

Need
to
decide
which
opAon
is
bener

•  What
are
the
alternaAves?

Expert
opinion
/

intuiAon

Historical
data

Why is experimentation better?
•  Our
intuiAon
is
not
very
good

(people
are
complex)

•  Historical
data

(becomes
outdated
fast)

What do we get if using A/B testing?
•  Reduced
design
discussions

•  Less
dependent
on
expert
opinions

•  Make
sure
all
changes
are
actually
(commercial)

improvements

•  AddiAonal
safety
net
for
new
changes
in
the
app

How complex is it?
if
(
newFeatureEnabled
==
true
)
{

//
new
feature
code

}
else
{

//
old
reliable
solu2on

}

+ add control from server-side
Today:

{

Experiment
A:
ON

Experiment
B:
ON

}

Server
Control experiments
At
any
moment
of
Ame:

{

Experiment
A:
ON

Experiment
B:
OFF

}

It’s easy! Can I start experimenting now?
Yes,
but
ﬁrst:

•  Pick
a
metric

•  Find
users
to
include
in
an
experiment

Metrics
Metrics
are
the
main
determinant
of
success!

1.  Pick
your
own
metric!

2.  Diversify
metrics!

Metrics
Metrics
are
the
main
determinant
of
success!

1.  Pick
your
own
metric!

2.  Diversify
metrics!

Metrics for Booking.com?

What is my metric?
Type
of
business
Possible
key
metrics

Ecommerce:
Booking.com,

Amazon

items
sold,
conversion,
loyalty

Content:
Euronews,
CNN
visitors,
loyalty,
engagement,
ad

clicks

Social
Networking
visitors,
engagement,
ad
clicks

Gaming:
Angry
birds
purchases,
subscripAons,
ad
clicks

Come
up
with

new
idea
to

improve
your

app

Run
an

experiment
to

see
which

variant
is
the

most
successful

Select
the

winner
based
on

metric

Remove
non-‐
winner
from
the

code

Experiment lifecycle

Tools for mobile that will allow you to start
experimenting today?
Yes,
just
Google
it!

For
example:

•  Airlock

•  Arise.io

•  AppAmize

•  Leanplum

…

A/B testing in apps provides
•  Final
changes
are
real
improvements

•  All
changes
are
server-‐side
controlled
and
can
be

enabled

disabled
at
any
point
of
Ame

A/B testing in apps provides
•  Final
changes
are
real
improvements

•  All
changes
are
server-‐side
controlled
and
can
be

enableddisabled
at
any
point
of
Ame

Is it enough for our apps?

Booking apps development
•  Use
Experiments

•  Develop
fast

•  Be
stable

•  Release
fast

•  Monitor
how
you
are
doing

Source: XKCD 303
http://imgs.xkcd.com/comics/compiling.png

Fast, iterative development

Android
–
slow,
>
3
min
to
build
on
big
projects

•  Buck
+
exopackage

•  Booking
DCL
(internal
soluAon)

•  others

iOS
–
usually
not
a
problem

Trunk development
What
&
how:

•  Everything
goes
into
master

•  Local
branches
up
to
a
developer

•  Tests
keep
master
stable

Why:

•  As
a
result
saves
Ame
on
merging

Making sure project builds and stable
•  Hooks

•  CI

•  TesAng

Hooks
Master
can
always
build!

git
push

Developer

pass

fail

git

repository

Hooks:

build
&
run

test

CI
git
push
git

repository

CI
system
Great
job

Developer

pass

fail

Email
&&
jabber
developer
about
the
problem

•  all
dev
build

•  special
builds

•  TranslaAons

•  UI
tests

Testing
•  Dedicated
tesAng
team

•  Easy
maintainable
tests
on
core
funcAonality
+
UI
tests

•  Bugbash
sessions

•  Internal
releases

•  3rd
party
tesAng

Release model
Train
release
model

•  Regular

•  Pre-‐planned
schedule

Release model
Train
release
model

•  Regular

•  Pre-‐planned
schedule

Nobody
waits:

you
take
this
train
or
the
next
one

Author: Maurits Vink
http://commons.wikimedia.org/wiki/File:VIRM6.jpg

Why train model?
•  Everyone
know
when
the
next
release
will
take
place

•  Team
can
plan
the
scope
to
be
done

•  Complexity
of
each
release
more
or
less
the
same

App roll out
Rolling
out
apps
not
as
fast
as
web:

Android:

•  ~1
day
to
all
users

iOS
:

•  Depends
on
Apple

•  Apple
review
process
1-‐5
days

App roll out
Rolling
out
apps
not
as
fast
as
web:

Android:

•  ~1
day
to
all
users

iOS
:

•  Depends
on
Apple

•  Apple
review
process
1-‐5
days

Hot fixes can’t be fast…
( Experiments can help! )

Safe app release
Android:

•  staged
rollouts

•  alpha
&
beta
tesAng

iOS:

•  Enterprise
signing
program

•  TestFlight
–
beta
tesAng
for
iOS
8

Staged rollout
Get
thing
done
right
for
Android
users

•  Make
sure
that
everything
goes
according
to
plan

•  NEW!
'Halt'
A
Staged
Rollout
if
found
a
problem

•  When
sure
-‐
release
to
100%

iOS:

•  No
support
for
staged
rollouts

Crash & data monitoring
Monitor
your
crashes,
errors
and
main
metrics
via:

•  CrashlyAcs,

•  Bugsense,

•  Google
AnalyAcs,

•  Flurry
AnalyAcs,

•  ….

Booking apps
ExperimenAng

•  Why
to
try

•  How

•  What
do
you
achieve

Developing

TesAng

Releasing

References
1.  Smartphone
Controlled
Paper
Airplane
hnps://www.kickstarter.com/projects/393053146/powerup-‐30-‐smartphone-‐controlled-‐
paper-‐airplane

2.  Jot:
CapaciAve
Touch
Stylus
by
Adonit!
hnps://www.kickstarter.com/projects/531383637/jot-‐capaciAve-‐touch-‐stylus/descripAon

3.  12
Surprising
A/B
Test
Results
to
Stop
You
Making
AssumpAons
hnp://unbounce.com/a-‐b-‐tesAng/shocking-‐results/

4.  5
Landing
Page
A/B
Tests
(And
Their
Surprising
Results)

hnp://blog.hubspot.com/markeAng/5-‐landing-‐page-‐a/b-‐tests-‐and-‐their-‐surprising-‐results

5.  Removing
Social
Sharing
Bunons
Increases
Conversions.
Yes,
You
Heard
That
Right!
hnps://vwo.com/blog/removing-‐social-‐sharing-‐bunons-‐from-‐ecommerce-‐product-‐page-‐
increase-‐conversions/

6.  Source:
XKCD
303
hnp://imgs.xkcd.com/comics/compiling.png

7.  Foto
van
twee
VIRM
VI
stellen
te
Amsterdam
Bijlmer
ArenA.
Author:
Maurits
Vink

hnp://commons.wikimedia.org/wiki/File:VIRM6.jpg

QuesAons?

yaroslav.wash

Iaroslav,

Khramov

Team
Leader
&

Android
developer

ya.khramov@gmail.com

Getting it Booking right

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Getting it Booking right

Similar to Getting it Booking right (20)

More from CodeFest

More from CodeFest (20)

Recently uploaded

Recently uploaded (20)

Getting it Booking right