2. INTRODUCTION TO AB TEST
This section contains theoretical information
about the AB test.
002
3. IN OTHER WORDS: INDEPENDENT TWO SAMPLE T TEST
It is used when it is desired to make a comparison between the
mean of two groups. In other words, if there are two main
population parameters whose true value is unknown, the
difference between these averages is to be examined and the
comparison is made over the samples obtained for this purpose.
3
4. WHAT ARE THE AB TEST IMPLEMENTATION STEPS?
◉ The hypothesis is established.
◉ The normality assumption is checked.
◉ The homogeneity of variance assumption is checked.
◉ If the assumptions are met, the parametric test is applied, if
not, the nonparametric test is applied.
4
5. LET’S START WITH THE ESTABLISHMENT OF THE HYPOTHESIS
FIRST!
◉ H0: µ1 = µ2
H1: µ1 ≠ µ2
5
◉ H0: µ1 >= µ2
H1: µ1 < µ2
◉ H0: µ1 <= µ2
H1: µ1 > µ2
The hypotheses of the AB test are as follows:
6. “
◉ H0: There is no statistically
significant difference between the
control group and the test group.
◉ H1: There is statistically significant
difference between the control
group and the test group.
6
7. WHAT DO THESE HYPOTHESIS DO?
Hypothesis testing is a statistical method used to test a belief.
Hypotheses are divided into two types among themselves: H0
and H1. The H0 hypothesis is called the 'hypothesis 0' and is
considered constant. As the 'alternative hypothesis', the H1
hypothesis comes into play.
7
8. “
H0 is rejected if the p-value is less
than 0.05. Otherwise it will not be
rejected.
8
9. THEN, WHAT IS P-VALUE?
In statistics, the p-value is a function used to measure how
extreme the observed sample results are based on a statistical
model.
If the model (hypothesis 0) is assumed to be true, the p-value is
the probability of obtaining values equal to or exceeding the
tested value.
9
10. “
If the p-value is less than the assumed
(acceptable) error amount α = 0.05,
the corresponding H0 hypothesis is
rejected.
10
11. LET'S CONTINUE WITH THE ASSUMPTION OF NORMALITY!
Shapiro-Wilks test is used for the assumption of normality and
it is decided whether the distribution is normal according to
the p-value.
11
12. “
◉ H0: There is no statistically significant
difference between the sample
distribution and the theoretical normal
distribution.
◉ H1: There is a statistically significant
difference between the sample
distribution and the theoretical normal
distribution.
12
13. NOW HOMOGENEOUS OF VARIANCE!
Levene test is used to examine the homogeneity of the
variances and it is decided whether the variances are
homogeneously distributed according to the p-value.
13
14. “
◉ H0: Variances are homogeneous.
◉ H1: Variances are not
homogeneous.
14
15. AND NOW AB TEST IS AVAILABLE!
The AB test can be applied to samples that pass the normality
assumption and variance homogeneity tests.
15
16. “
◉ ttest_ind test is applied when the
assumptions are satisfied.
◉ mannwhitneyu nonparametric test
is applied if the assumptions are
not satisfied.
16
17. 01
-----
UNDERSTAND THE WORK
Sharing of the problem
02
-----
UNDERSTAND THE DATA
Basic information about
the dataset
03
-----
AB TESTING
Application of AB test after
assumption tests
17
19. PROBLEM
abc.veribilimiokulu.com has the X suggestion system and they
received an offer to switch to the Y suggestion system.
They have decided to test the Y recommendation system and
want to do an AB test to see if it has more payoffs.
19
21. THE STORY OF DATASET
The data set contains information on clicks and purchases
separately for abc.veribilimiokulu.com's X recommendation
system and Y recommendation system.
21
22. VARIABLES
22
VARIABLE NAME DESCRIPTION VARIABLE TYPE
Impression Ad views Numeric
Clicks Indicates the number
of clicks on the
displayed ad.
Numeric
Purchase Indicates the number
of products purchased
after the ads clicked.
Numeric
Earning Earnings after
purchased products
Numeric
25. NORMALITY
p-value for the control group is greater than 0.05. ‘’H0: There is
no statistically significant difference between the sample
distribution and the theoretical normal distribution.’’ hypothesis
is not rejected. Distribution is normal.
25
27. NORMALITY
p-value for the test group is greater than 0.05. ‘’H0: There is no
statistically significant difference between the sample
distribution and the theoretical normal distribution.’’ hypothesis
is not rejected. Distribution is normal.
27
29. HOMOGENCY OF VARIANCE
p-value is greater than 0.05. ‘’ H0: Variances are homogeneous. ’’
hypothesis is not rejected. Parametric AB Test can be applied.
29
31. APPLICATION OF THE AB TEST
p-value is greater than 0.05. ‘’H0: There is no statistically
significant difference between the control group and the test
group.’’ hypothesis is not rejected.
31
32. INTERPRETATION OF AB TEST RESULT
There is no statistically significant difference between the
returns of these two recommendation systems. Therefore, it
would be unnecessary to allocate a budget to the new proposal
system as there is no difference. It can be continued with the
existing suggestion system.
32
