Presentation at the 21th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 2015). ABSTRACT: Online controlled experiments are widely used to improve the performance of websites by comparison of user behavior related to different variations of the given website. Although such experiments might have an important effect on the key metrics to maximize, small-scale websites have difficulty applying this methodology because they have few users. Furthermore, the candidate variations increase exponentially with the number of elements that must be optimized. A testing method that finds a high-performing variation with a few samples must be devised to address these problems. As described herein, we formalize this problem as a website optimization problem and provide a basis to apply existing search algorithms to this problem. We further organize existing testing methods and extract devices to make the experiments more effective. By combining organized algorithms and devices, we propose a rapid testing method that detects high-performing variations with few users. We evaluated our proposed method using simulation experiments. Results show that it outperforms existing methods at any website scale. Moreover, we implemented our proposed method as an optimizer program and used it on an actual small-scale website. Results show that our proposed method can achieve 57% higher performance variation than that of the generally used A/B testing method. Therefore, our proposed method can optimize a website with fewer samples. The website optimization problem has broad application possibilities that are applicable not only to websites but also to manufactured goods.

1. Website Optimization
Problem and Its Solutions
Shuhei Iitsuka and Yutaka Matsuo
The University of Tokyo

2. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
A/B testing is powerful.
2
ref. How Obama Raised $60 million by Running a Simple Experiment
http://blog.optimizely.com/2010/11/29/how-obama-raised-60-million-by-running-a-simple-experiment/
8.3% 11.6%sign-up rate
$60M!

3. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Sample size is power.
3
Result
Result

4. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
See the wood first.
4
See the wood first. Search the neighbors.
Initialization Phase Local Search Phase

5. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Agenda
1. Related Studies
2. Website Optimization Problem
3. Proposed Testing Method
4. Experimental Results
5. Discussion & Conclusion
5

6. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Past Studies
Giants making profits by online testing with a large number of users.
6
1. Related Studies
However, how can we use it for smaller websites?

7. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Existing Testing Methods
7
A B
A B
A B
A B
B
A
A
B
A B
A B
A B
A B
A B
A/B Testing Full Factorial Design
Fractional Factorial Design Bandit Algorithm
1. Related Studies

8. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Agenda
1. Related Studies
2. Website Optimization Problem
3. Proposed Testing Method
4. Experimental Results
5. Discussion & Conclusion
8

9. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Expression of a Variation
A website variation can be denoted as a combination of elements.
9
=( , , )
Variation
→ The problem can be defined as a combinatorial optimization problem.
“GET
INVOLVED”
“CHANGE”
2. Website Optimization Problem
Website Variation:
Page Element:
x = (x1, · · · , xm)
xi 2 Vi

10. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Interaction with Users
10
p(y|x)f(x) ' E[y|x]
The evaluation value need to be estimated from the given feedback.
y ⇠ p(y|x)f(x) ' E[y|x] where
→ The evaluation function is estimated by the expected value.
2. Website Optimization Problem
xWebsite Variation
yUser Behavior

11. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Website Optimization Problem
Find the solution which satisfies the following equation.
11
x⇤
= arg max
x2X
E[y|x] s.t. y ⇠ p(y|x)
• maximizes the conditional expected value of the key metrics.
• is derived from the probability distribution.
2. Website Optimization Problem
x⇤
y
x⇤

12. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Local Search Solution
12
1. Initialization
2. Repeat until no improvement is made or all samples have been used.
2-1. Neighbor Solution Generation
2-2. Solution Move
X
x 2 X
X0
Neighbors(x)
x Move(x, X0
)
2. Website Optimization Problem

13. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Agenda
1. Related Studies
2. Website Optimization Problem
3. Proposed Testing Method
4. Experimental Results
5. Discussion & Conclusion
13

14. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Organization of Existing Testing Methods
14
Search Algorithm Technique
A/B Testing Local Search None
Full
Factorial Design
Brute-force Search None
Fractional
Factorial Design
Brute-force Search Linear Assumption
Bandit Algorithm Brute-force Search
Flexible Sample
Allocation
3. Proposed Testing Method

15. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Technique #1: Linear Assumption
15
Color Label Location
A B C L R
x = (x1, x2, x3)
3. Proposed Testing Method

16. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Technique #2: Flexible Sample Allocation
16
3. Proposed Testing Method
3.2% 2.4% 5.6% 1.6%
Expected Value

17. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Racing Algorithm
Another implementation of Flexible Sample Allocation.
17
3. Proposed Testing Method
ClickThroughRate
A B C D E
Variation
Remove
Adopt

18. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Overview of Proposed Method
18
Initialization Phase Local Search Phase
• Collects data from the entire
solution space.
• Estimates the optimal solution
with linear assumption.
• Start Local Search starting from
the estimated solution.
3. Proposed Testing Method
+ streamlined by flexible sample allocation

19. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Agenda
1. Related Studies
2. Website Optimization Problem
3. Proposed Testing Method
4. Experimental Results
5. Discussion & Conclusion
19

20. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Evaluation Experiments
1. Simulation Experiment / Artificial Problem
2. Simulation Experiment / Actual Large-scale Website
3. Practical Experiment / Actual Small-scale Website
20
4. Experimental Results

21. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Testing Methods
21
Method Initialization Local Search
BF
(Brute-force)
Random N/A
LA
(Linear Assumption)
Linear Assumption N/A
LS
(Local Search)
Random Local Search
LALS
(Linear Assumption +
Local Search)
Linear Assumption Local Search
LALS+
(LALS +
Racing Algorithm)
Linear Assumption +
Flexible Allocation
Local Search +
Flexible Allocation
Baseline
Proposal
4. Experimental Results

22. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Exp. #1: Simulation on Artificial Problems
22
Problem Evaluation Function Sample Size
#1 Linear Init. Only
#2 Linear Init. + Local Search
#3 Non-Linear Init. Only
#4 Non-Linear Init. + Local Search
f2(x) = x1 + x2 + x3 x4 x5 x6 x1x2 + N(0, 1)
f1(x) = x1 + x2 + x3 x4 x5 x6 + N(0, 1)
Problem Settings
Linear Evaluate Function
Non-Linear Evaluate Function
Nf(x)
xi 2 {0, 1, 2}
4. Experimental Results
Non-Linear Member
Noise

23. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Each method is evaluated by the accuracy of the estimated optimal solution.
23
Exp. #1 Results
Problem BF LA LS LALS LALS+
#1 (Linear/Small) 0.24 1.00 0.00 1.00 1.00
#2 (Linear/Large) 0.54 1.00 0.01 1.00 1.00
#3 (Non-Linear/Small) 0.26 0.14 0.01 0.22 0.22
#4 (Non-Linear/Large) 0.46 0.26 0.02 0.33 0.68
Baseline Proposal
Linear assumption works well with
the linear evaluation function.
Flexible sample allocation
boosts the local search.
4. Experimental Results

24. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Exp. #2: Simulation on a Large Website
• Actual large-scale website with 1000-10000 visiters/day.
• Key metrics: Ads Click-through Rate
• Evaluation function is simulated from the log (Mar 14-22, 2013)
24
A B C
Which one does maximize CTR?
SPYSEE http://spysee.jp
q(x) = 0.0640 + 0.0117xA 0.0067xB 0.0134xC
xi 2 {0, 1} (Apply the change or not)
4. Experimental Results

25. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Exp. #2 Results
25
0.25
0.50
0.75
1.00
0 10000 20000 30000
Sample Size n
Accuracy
Method
LALS+
LALS
LS
LA
BF
Average accuracy of each algorithm LA exhibits the best performance
because the evaluation function
is linear.
Our proposed methods succeeds
to start the local search from the
promising initial solution.
LALS+ can improve the
performance rapidly with the
flexible sample allocation.
Init. Local Search
4. Experimental Results

26. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Exp. #3: Practical Test on a Small Website
• Implemented our proposed method as an optimizer program.
• Actual small-scale website with hundreds of visitors/day.
• LS (Baseline) VS. LALS (Proposal)
• Key metric: Page views per session
26
Element Values
Thumbnail border width 0px, 5px
Thumbnail margin 0px, 5px, 10px
Thumbnail Size 100px, 200px, 300px
Thumbnail Shape square, circle
Imagerous* http://imagero.us
Tested Elements
4. Experimental Results

27. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Exp. #3: Results
• LALS reached a 57% higher
solution.
(t-test: 99% confidence)
• Our proposed method
functions as a practical
optimizer program with an
actual small-scale website.
27
Transition of the current solution and
the expected value.
ExpectedValueE[y|x]
0
2
4
6
8
Sample Size n
0 175 350 525 700
LS
LALS
4. Experimental Results
57%

28. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Agenda
1. Related Studies
2. Website Optimization Problem
3. Proposed Testing Method
4. Experimental Results
5. Discussion & Conclusion
28

29. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
From Bits to Atoms
29
Requirements
Each solution is expressed as
a combination of elements.
Reconfiguration cost is zero.
ex.) 3D printers
User feedback is observable.
ex.) Review website
5. Discussion & Conclusion

30. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Conclusion
• We formalized existing testing methods and a website
optimization problem.
• We proposed a new rapid testing method which works on small-
scale websites.
• We evaluated that our proposed method works on actual small-
scale websites.
30
5. Discussion & Conclusion

31. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Future Works
31
Make a Hypothesis
x 2 X
Define Metrics
f(x)
Explore the Solution
x⇤
= arg max
x2X
f(x)
We’ve tackled this!
Which key metrics we
need to focus for
effective experiments?
How do we define our
website as a set of
variables?
How can we automate
the generation of
candidates?
Website Optimization Process
5. Discussion & Conclusion

32. “Website Optimization Problem and Its Solutions (Paper ID:516)” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo
Shuhei Iitsuka, The University of Tokyo.
tushuhei.com
iitsuka@weblab.t.u-tokyo.ac.jp
Thank you for listening.
32

33. “Website Optimization Problem and Its Solutions (Paper ID:516)” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo
Appendix
33

34. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
X: Candidate Solutions
Y ← {} : Empty Set for Observed Data, n ← 0 : Number of Observations.
N_1: Sample Size for Initialization Phase, N_2: Sample Size for Local Search Phase.
FOR N_1 TIMES:
Y ← Observe(RandomChoice(X))
n++
x* ← LinearAssumption(Y)
WHILE n < N DO:
x’ ← GetNeighborSolution(x*, X)
FOR N_2 TIMES:
Y ← Observe(x’)
n++
x* ← Update(x*, x’, Y)
RETURN x*
34
Initialization
Local Search
3. Proposed Testing Method
+ Streamlined by
flexible allocation

35. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
DOE and Linear Assumption
• DOE (Design of Experiment) is used in traditional industries
which have huge cost to reconfigure the environment.
• Websites require no cost to change the parameters.
→ We can conduct random observation, then apply ANOVA to
estimate each element’s effect.
35
Design of Experiment:
Design beforehand.
Linear Assumption:
Random collection first.
Zero
Reconfiguration Cost
5. Discussion & Conclusion

36. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Webpage Segmentation
36
1ϕ
2ϕ
3ϕ
1
2
ϕ 2
2
ϕ
(a) (b)Cai, Deng, et al. Vips: a vision-based page segmentation algorithm. Microsoft technical report, MSR-TR-2003-79,
2003.

37. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Page Element Extraction
37
WELCOME!
JOIN NOW
Background: {WHITE, BLACK}
Button Color: {WHITE, BLACK}
Strong Interactive
Effect?

38. “Website Optimization Problem and Its Solutions” Shuhei Iitsuka and Matsuo Yutaka, The University of Tokyo. KDD 2015.
Bandit Algorithm
• ε-greedy
• ε: exploration, 1 - ε: exploitation
• Softmax: High expected value → High exploitation rate
• UCB1: Expected value + Freshness bonus
38
解 x をユーザに表示する確率 p(x) は式 2.1 によって表される。
p(x) =
exp(yx/τ)
x∈X exp(yx/τ)
(2.1)
呼ばれるパラメータであり、探究心の強さを表している。温度が非常に高い
わち τ → ∞ のときは解 x を選ぶ確率 p(x) は 1/|X| に収束するため、すべ
等の確率で選ばれることになる。逆に温度が低いときは yx が効き始めるた
価値の期待値が高い解が 1 に近い確率で選ばれるようになる。
CB1 ではこれまでに紹介したアルゴリズムとは異なり、ランダム性を用い
1 では基本的に評価値の期待値 yx が最も高い解を選ぶ戦略だが、解を選ん
じてボーナスが追加される。解 x ∈ X を表示した回数を tx とすると、解 x
ux は
ux = yx +
2 log( x∈X tx)
tx
、この UCB 値を最大にする解 x が選択される。
い解を優先的に表示することで実験による損失を免れている。簡単に実
、解の期待値に関わらず探求または活用を選択するため、期待値に大き
でも期待値が低い解を選んでしまう可能性がある。
max アルゴリズムでは、解の評価値の期待値に応じて表示する確率を
空間を X、観測データから算出される解 x ∈ X の評価値の期待値を yx
をユーザに表示する確率 p(x) は式 2.1 によって表される。
p(x) =
exp(yx/τ)
x∈X exp(yx/τ)
(2.1)
れるパラメータであり、探究心の強さを表している。温度が非常に高い
τ → ∞ のときは解 x を選ぶ確率 p(x) は 1/|X| に収束するため、すべ
確率で選ばれることになる。逆に温度が低いときは yx が効き始めるた
の期待値が高い解が 1 に近い確率で選ばれるようになる。
ではこれまでに紹介したアルゴリズムとは異なり、ランダム性を用い