Spark Summit East Talk

3. I listen to ~ 100 Bln ad opportunities daily

4. I listen to ~ 100 Bln ad opportunities daily
I respond with optimal bids within milliseconds

5. I listen to ~ 100 Bln ad opportunities daily
I respond with optimal bids within milliseconds
I petabytes of data (ad impressions, visits, clicks, conversions)

6. Predicting user response to ads is a Machine-Learning problem.

7. Predicting user response to ads is a Machine-Learning problem.
but quantifying impact of ad-exposure is a Measurement probem.

8. Spark: existing vs simulated data
Most Spark applications process existing big data-sets.

9. Spark: existing vs simulated data
Most Spark applications process existing big data-sets.
Today we’re talking about analyzing simulated big data

10. Key Conceptual Take-aways
I Issues in Ad lift measurement

11. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition

12. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds

13. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds

14. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds
I Gibbs Sampling (MCMC – Markov Chain Monte Carlo)

15. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds
I Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
I Using Spark for:

16. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds
I Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
I Using Spark for:
I Monte Carlo sampling for confidence-bounds

17. Key Conceptual Take-aways
I Issues in Ad lift measurement
I Proper definition
I Confidence bounds
I Bayesian Methods for Ad Lift Confidence Bounds
I Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
I Using Spark for:
I Monte Carlo sampling for confidence-bounds
I Monte Carlo simulations

18. Application context: ad impact measurement
I Advertisers want to know the impact of showing ads to users.

19. Measuring Ad Impact: Two Approaches
I Observational studies:

20. Measuring Ad Impact: Two Approaches
I Observational studies:
I Compare uses who happen to be exposed vs not exposed

21. Measuring Ad Impact: Two Approaches
I Observational studies:
I Compare uses who happen to be exposed vs not exposed
I Bias a big issue

22. Measuring Ad Impact: Two Approaches
I Observational studies:
I Compare uses who happen to be exposed vs not exposed
I Bias a big issue
I Randomized tests:

23. Measuring Ad Impact: Two Approaches
I Observational studies:
I Compare uses who happen to be exposed vs not exposed
I Bias a big issue
I Randomized tests:
I Randomly expose to test, compare with control (un-exposed)

24. Ideal Randomized Test

25. Ideal Randomized Test

26. Ideal Randomized Test

27. Ideal Randomized Test: Ad lift

28. Ideal Randomized Test: Ad lift

29. Ad Lift: Response Rates
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?

30. Ad Lift: Response Rates
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?
Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .

31. Ad Lift: Response Rates
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?
Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .
But how confident are we?

32. Response Rate 90% Confidence Bounds

33. Response Rate 90% Confidence Bounds
P(R > ˆR | r = q5) = 5%

34. Response Rate 90% Confidence Bounds
P(R > ˆR | r = q5) = 5%
P(R < ˆR | r = q95) = 5%

35. Response-Rate Confidence Bounds

36. Response-Rate Confidence Bounds

37. Response-Rate Confidence Bounds

38. Response-Rate Confidence Bounds
How to find (q5, q95) ?

40. Response-Rate: Bayesian Confidence Bounds
Randomly generate response rates that are consistent with the data.

41. Response-Rate: Bayesian Confidence Bounds
Randomly generate response rates that are consistent with the data.
(Sample rates from posterior distribution given data.)

42. Response-Rate: Bayesian Confidence Bounds
Randomly generate response rates that are consistent with the data.
(Sample rates from posterior distribution given data.)
Find the (0.05, 0.95) quantiles of these rates.

43. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)

44. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)

45. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)
à rk
(1 ≠ r)N≠k
· Beta(1, 1)

46. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)
à rk
(1 ≠ r)N≠k
· Beta(1, 1)
à rk+1
(1 ≠ r)N≠k+1

47. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)
à rk
(1 ≠ r)N≠k
· Beta(1, 1)
à rk+1
(1 ≠ r)N≠k+1
à Beta(k + 1, N ≠ k + 1)

48. Response-Rate: Bayesian Confidence Bounds
I Assume an unknown true rate r, with a prior distrib. p(r)
I assume p(r) = Beta(1, 1) = Unif (0, 1)
I Sample from the posterior distribution of the rate r
I conditional on the observed data (k conversions out of N)
P(r | k) Ã P(k | r) · p(r)
à rk
(1 ≠ r)N≠k
· Beta(1, 1)
à rk+1
(1 ≠ r)N≠k+1
à Beta(k + 1, N ≠ k + 1)
I Compute (0.05, 0.95) quantiles from the generated rates.

49. Response-Rate: Bayesian Confidence Bounds
A simple form of Gibbs Sampling (more later):
I sample M values of r from posterior
P(r | k) ≥ Beta(k + 1, N ≠ k + 1).
I compute (0.05, 0.95) quantiles

50. Response-Rate: Bayesian Confidence Bounds
A simple form of Gibbs Sampling (more later):
I sample M values of r from posterior
P(r | k) ≥ Beta(k + 1, N ≠ k + 1).
I compute (0.05, 0.95) quantiles
from numpy.random import beta
from scipy.stats.mstats import mquantiles
def conf(N, k, samples = 500):
rates = beta(k+1, N-k+1, samples)
return mquantiles(rates, prob = [0.05, 0.95])

51. Response-Rate: Bayesian Confidence Bounds

52. Response-Rate: Bayesian Confidence Bounds

53. Response-Rate: Bayesian Confidence Bounds

54. Response-Rate: Bayesian Confidence Bounds

55. Response-Rate: Bayesian Confidence Bounds

56. Response Rates: Example
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?
Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .

57. Response Rates: Example
If we see k = 200 conversions out of N = 10, 000 users,
what is a good estimate for the response-rate?
Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .
=∆ 90% confidence region (1.8%, 2.2%)

58. We’ve talked about Response Rates. . .
now let’s consider Ad Lift

59. Ad Lift: Simple Example
I control: 10,000 users, 200 conversions
I test: 100,000 users, 2200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%

60. Ad Lift: Simple Example
I control: 10,000 users, 200 conversions
I test: 100,000 users, 2200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
This is a great lift !

61. Ad Lift: Simple Example
I control: 10,000 users, 200 conversions
I test: 100,000 users, 2200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
This is a great lift !
Not so fast! Is this a reliable estimate?

62. Ad Lift: Simple Example
I control: 10,000 users, 200 conversions
I test: 100,000 users, 2200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
This is a great lift !
Not so fast! Is this a reliable estimate?
Could true lift ¸ be 0%, or even negative ?

63. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:

64. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:
I draw control response rate rc from posterior
P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1).

65. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:
I draw control response rate rc from posterior
P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1).
I draw test response rate rt from posterior
P(rt | kt) ≥ Beta(kt + 1, Nt ≠ kt + 1).

66. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:
I draw control response rate rc from posterior
P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1).
I draw test response rate rt from posterior
P(rt | kt) ≥ Beta(kt + 1, Nt ≠ kt + 1).
I compute lift L = rt/rc ≠ 1

67. Ad Lift: Bayesian Confidence Bounds
Sampling approach:
Observed data: control: (kc, Nc), test: (kt, Nt)
1. Repeat M times:
I draw control response rate rc from posterior
P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1).
I draw test response rate rt from posterior
P(rt | kt) ≥ Beta(kt + 1, Nt ≠ kt + 1).
I compute lift L = rt/rc ≠ 1
2. Compute (0.05, 0.95) quantiles of set of M lifts {L}.

68. Ad Lift: Bayesian Confidence Intervals
I control: nc = 10, 000 users, kc = 200 conversions
I test: nt = 100, 000 users, kt = 2, 200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%

69. Ad Lift: Bayesian Confidence Intervals
I control: nc = 10, 000 users, kc = 200 conversions
I test: nt = 100, 000 users, kt = 2, 200 conversions
Observed response-rates:
I control: ˆRc = 200/10, 000 = 2%
I test: ˆRt = 2200/100, 000 = 2.2%
Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
90% confidence interval: (≠2.7%, 23.6%)

71. Complication 1:
Auction win-bias

72. Ideal Randomized Test

73. Ideal Randomized Test

74. Ideal Randomized Test

75. Ideal Randomized Test
Bids on control users are wasted!

77. A Less Wasteful Randomized Test

78. A Less Wasteful Randomized Test: Win-bias
Cannot simply compare Test Winners (tw) and Control (c):
I test-winners selection bias: “win bias”

79. Ad Lift: Proper Definition

80. Ad Lift: Proper Definition

81. Ad Lift: Proper Definition

82. Ad Lift: Proper Definition

83. Ad Lift: Proper Definition

84. Ad Lift: Proper Definition

85. Ad Lift Estimation
Main ideas:
I observe test-losers response rate RtL

86. Ad Lift Estimation
Main ideas:
I observe test-losers response rate RtL
I observe test win-rate w

87. Ad Lift Estimation
Main ideas:
I observe test-losers response rate RtL
I observe test win-rate w
I we show one can estimate
R0
tw =
Rc ≠ (1 ≠ w)RtL
w

88. Ad Lift Estimation
Main ideas:
I observe test-losers response rate RtL
I observe test win-rate w
I we show one can estimate
R0
tw =
Rc ≠ (1 ≠ w)RtL
w
I compute lift L = R1
tw /R0
tw ≠ 1

89. Ad Lift Estimation
Main ideas:
I observe test-losers response rate RtL
I observe test win-rate w
I we show one can estimate
R0
tw =
Rc ≠ (1 ≠ w)RtL
w
I compute lift L = R1
tw /R0
tw ≠ 1
I similar to Treatment E ect Under Non-compliance in clinicial
trials.

90. Ad Lift Estimation
How to compute the 90% confidence interval for L?

92. Ad Lift: Confidence Intervals with Gibbs sampler

93. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:

94. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors

95. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities

96. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities
I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet)

97. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities
I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet)
I Sample M values of unknown ◊ from posterior: Gibbs Sampler
P(◊ |Data) Ã P(Data | ◊) · p(◊)

98. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities
I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet)
I Sample M values of unknown ◊ from posterior: Gibbs Sampler
P(◊ |Data) Ã P(Data | ◊) · p(◊)
I For each sampled ◊ compute lift L using above

99. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach (details omitted, see Chickering/Pearl 1997):
I Assume a random parameter vector ◊ consisting of:
I user latent (potential) behaviors
I their probabilities
I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet)
I Sample M values of unknown ◊ from posterior: Gibbs Sampler
P(◊ |Data) Ã P(Data | ◊) · p(◊)
I For each sampled ◊ compute lift L using above
I Compute (0.05, 0.95) quantiles of sampled L values

100. Ad Lift: Confidence Intervals

101. Ad Lift: Confidence Intervals
Gibbs sampler convergence may depend on prior distribution:
I start with multiple (say 100) priors
I run them all in parallel using Spark.

102. Uses of Monte Carlo Simulations
I confidence intervals

103. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating

104. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating
I response rates

105. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating
I response rates
I lift

106. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating
I response rates
I lift
I understand e ect of complex phenomena

107. Uses of Monte Carlo Simulations
I confidence intervals
I determine “su cient” population sizes for reliably estimating
I response rates
I lift
I understand e ect of complex phenomena
I validate/verify analytical formulas

108. Complication 2:
Control contamination due to users with multiple cookies

110. Control Contamination due to Multiple Cookies

111. Control Contamination due to Multiple Cookies

112. Control Contamination due to Multiple Cookies

113. Control Contamination due to Multiple Cookies

114. Control Contamination due to Multiple Cookies

115. Cookie-Contamination Questions
I How does cookie contamination a ect measured lift?

116. Cookie-Contamination Questions
I How does cookie contamination a ect measured lift?
I Does the cookie-distribution matter?

117. Cookie-Contamination Questions
I How does cookie contamination a ect measured lift?
I Does the cookie-distribution matter?
I everyone has k cookies vs an average of k cookies

118. Cookie-Contamination Questions
I How does cookie contamination a ect measured lift?
I Does the cookie-distribution matter?
I everyone has k cookies vs an average of k cookies
I What is the influence of the control percentage?

119. Cookie-Contamination Questions
I How does cookie contamination a ect measured lift?
I Does the cookie-distribution matter?
I everyone has k cookies vs an average of k cookies
I What is the influence of the control percentage?
I Simulations best way to understand this

120. Simulations for cookie-contamination
I A scenario is a combination of parameters:
I M = # trials for this scenario, usually 10K-1M
I n = # users, typically 10K - 10M
I p = # control percentage (usually 10-50%)
I k = cookie-distribution, expressed as 1 : 100, or 1 : 70, 3 : 30
I r = (un-contaminated) control user response rate
I a = true lift, i.e. exposed user response rate = r ú (1 + a).
I A scenario file specifies a scenario in each row.
I could be thousands of scenarios

121. Scenario Simulations in Spark

122. Scenario Simulations in Spark

123. Scenario Simulations in Spark

124. Scenario Simulations in Spark

125. Scenario Simulations in Spark