Identifying causal effects is an integral part of scientific inquiry, spanning a wide range of questions such as understanding behavior in online systems, effect of social policies, or risk factors for diseases. In the absence of a randomized experiment, however, traditional methods such as matching or instrumental variables fail to provide robust estimates because they depend on strong assumptions that are never tested.   My research shows that many of the strong assumptions are testable. This leads to a data mining framework for causal inference from observed data: instead of relying on untestable assumptions, we develop tests for valid experiment-like data---a "natural" experiment---and estimate causal effects only from subsets of data that pass those tests. Two such methods are presented. The first utilizes auxiliary data from large-scale systems to automate the search for natural experiments. Applying it to estimate the additional activity caused by Amazon's recommendation system, I find over 20,000 natural experiments, an order of magnitude more than those in past work. These experiments indicate that less than half of the click-throughs typically attributed to the recommendation system are causal; the rest would have happened anyways. The second method proposes a general Bayesian test that can be used for validating natural experiments in any dataset. For instance, I find that a majority of natural experiments used in recent studies in a premier economics journal are likely invalid. More generally, the proposed framework presents a viable way of doing causal inference in large-scale datasets with minimal assumptions.

1. Causal data mining:
Identifying causal
effects at scale
1
AMIT SHARMA
Postdoctoral Researcher, Microsoft Research New York
http://www.amitsharma.in
@amt_shrma

2. 2

3. 3

5. Distinguishing between personal preference and homophily
in online activity feeds. Sharma and Cosley (2016).
Studying and modeling the effect of social explanations
in recommender systems. Sharma and Cosley (2013).
Amit and Dan like this.

6. Distinguishing between personal preference and homophily
in online activity feeds. Sharma and Cosley (2016).
Studying and modeling the effect of social explanations
in recommender systems. Sharma and Cosley (2013).
Amit and Dan like this.
Averaging Gone Wrong: Using Time-Aware Analyses to Better
Understand Behavior. Barbosa, Cosley, Sharma, Cesar (2016)
Auditing search engines for differential satisfaction across
demographics. Mehrotra, Anderson, Diaz, Sharma, Wallach (2016)

7. 7
Jake and Duncan like this

8. 8

9. 9
Cause (X=𝑥) Outcome (Y)
Unobserved
Confounders (U)
𝒚 = 𝑓 𝒙, 𝑢
𝒙 = 𝑔 𝑢

10. 10

11. 11

12. 12

14. 14

15. 15

16. 16
Data mining for causal inference

17. Part 0: Traditional causal
inference using a natural
experiment
17

18. 1854: London was having a devastating
cholera outbreak 18

19. Causal question: What
is causing cholera?
Air-borne: Spreads
through air (“miasma”)
Water-borne: Spreads
through contaminated
water
19

20. Polluted Air
Cholera
Diagnosis
Contaminated
Water
Cholera
Diagnosis
Neighborhood

21. 21
Enter John Snow. He found higher cholera deaths near
a water pump, but could be just correlational.

22. 22
New Idea: Two major water companies for London:
one upstream and one downstream.

23. 23
No difference in neighborhood, still an 8-fold increase
in cholera with the downstream company.
S&V
and
Lambeth

24. Led to a change in belief about cholera’s cause.
24

25. 25

26. 26
Contaminated
Water
Cholera
Diagnosis
Neighborhood
Water
Company

27. 27
Contaminated
Water (X)
Cholera
Diagnosis
(Y)
Other factors
[e.g. neighborhood]
(U)
Water
Company
(Z)
As-If-Random
Exclusion

28. 28
Cause (X) Outcome (Y)
Unobserved
Confounders
(U)
New
variable (Z)
As-If-Random
Exclusion

29. 29
Cause
(X)
Outcome
(Y)
Unobserved
Confounders
(U)
Randomized
Assignment
(Z)

30. Cause
(X)
Outcome
(Y)
Unobserved
Confounders
(U)
Instrumental
Variable
(Z)
30

32. 32
Such that:
As-If-Random:
Exclusion: 𝑍 ∐ 𝑌 | 𝑋, 𝑈

33. 33

34. 34

35. Part I: Split-door criterion for
causal identification
35

36. 36
Cause Outcome
Unobserved
Confounders
Auxiliary
Outcome

37. 37
Cause
Primary
Outcome
Unobserved
Confounders
Auxiliary
Outcome

38. Cause Outcome
Unobserved
Confounders
Auxiliary
Outcome

39. Cause Outcome
Unobserved
Confounders
Auxiliary
Outcome

40. 40

41. 41

42. 42

43. 43

44. 44

45. 45
Demand for
The Road
Visits to The
Road
Rec. visits to
No Country
for Old Men
Demand for
No Country
for Old Men

46. 46
Causal
Convenience
OBSERVED ACTIVITY
FROM RECOMMENDER
All page
visits
?
ACTIVITY WITHOUT
RECOMMENDER

47. 47

48. 48
Treatment (A) Control (B)

49. 49
Instrument
Demand for
Cormac McCarthy
Visits to The
Road
Rec. visits to
No Country
for Old Men

50. 50

51. 51
All visits to a
recommended product
Recommender
visits
Direct visits
Search visits
Direct
browsing
Auxiliary outcome: Proxy for
unobserved demand

52. 52
Demand for
focal product
(UX)
Visits to
focal product
(X)
Rec. visits
(YR)
Direct visits
(YD)
Demand for
rec. product
(UY)

53. ? ?
53

54. 54

55. 55
Criterion: 𝑿 ∐ 𝒀 𝑫
Demand for
focal product
(UX)
Visits to
focal product
(X)
Rec. visits
(YR)
Direct visits
(YD)
Demand for
rec. product
(UY)

56. 56
Unobserved
variables (UX)
Cause
(X)
Outcome
(YR)
Auxiliary
Outcome
(YD)
Unobserved
variables (UY)

57. 57
𝑦𝑟 = 𝑓 𝑥, 𝑢 = 𝛾2 𝑢 + 𝝆𝒙 + 𝜖 𝑦𝑟
𝑥 = 𝑔 𝑢 = 𝛾1 𝑢 + 𝜖 𝑥
𝑦 𝑑 = ℎ 𝑢 = 𝛾3 𝑢 + 𝜖 𝑦𝑑

58. 58
Independence test used to
find natural experiments.
Only Assumption: Auxiliary
outcome is affected by the
causes of the primary
outcome.

59. 59
Treatment Outcome
Unobserved
Confounders
Instrumental
Variable
Treatment Outcome
Unobserved
Confounders
Auxiliary
Outcome

61. 61

62. Recreating sequence of visits: Log data
62
Timestamp URL
2014-01-20
09:04:10
http://www.amazon.com/s/ref=nb_sb_noss_1?fiel
d-keywords=Cormac%20McCarthy
2014-01-20
09:04:15
http://www.amazon.com/dp/0812984250/ref=sr_
1_2
2014-01-20
09:05:01
http://www.amazon.com/dp/1573225797/ref=pd
_sim_b_1

63. Recreating sequence of visits: Log data
63
Timestamp URL
2014-01-20
09:04:10
http://www.amazon.com/s/ref=nb_s
b_noss_1?field-
keywords=Cormac%20McCarthy
2014-01-20
09:04:15
http://www.amazon.com/dp/081298
4250/ref=sr_1_2
2014-01-20
09:05:01
http://www.amazon.com/dp/157322
5797/ref=pd_sim_b_1
User searches for
Cormac McCarthy
User clicks on the
second search result
User clicks on the first
recommendation

65. 65

67. 67
𝑡 = 15 days

68. Compute 𝜌 = 𝑌𝑅/ 𝑋
68

69. 69

70. 70

71. 71

72. 72

73. 73

74. 74

75. 75

76. 76
Lower CTR may be due
to the holiday season

77. 77

78. 78

79. 79
Oprah [Carmi et al.] 133 shocks Restricted to books
Split-door criterion 12,000 natural
experiments
Representative of overall
product distribution

80. 80

81. Part 2: A general Bayesian
test for natural experiments
in any dataset
81

82. Cause
(X)
Outcome
(Y)
Unobserved
Confounders
(U)
Instrumental
Variable
(Z)
As-If-Random?
Exclusion?

83. 83
Cause
(X)
Outcome
(Y)
Unobserved
Confounders
(U)
I.V.
(Z)
(X) (Y)
(U)
(Z)
(X) (Y)
(U)
(Z)
(X) (Y)
(U)
(Z)

84. 84
Cause
(X)
Outcome
(Y)
Unobserved
Confounders
(U)
I.V.
(Z)

85. 85

86. 86

87. 87
Both Valid and
Invalid IV models
can generate this
data distribution.
Can attain a weaker notion: probable sufficiency

88. 88

89. 89
(X) (Y)
(U)
(Z)(X) (Y)
(U)
(Z)
(X) (Y)
(U)
(Z)(X) (Y)
(U)
(Z)

90. 90
Data is likely to be generated from a Valid-IV model if
ValidityRatio ≫ 1

91. 91

92. 92
𝑦 =
0 ∶ (𝑓𝑎)
𝑥 ∶ (𝑓𝑏)
~𝑥 ∶ (𝑓𝑐)
1 ∶ (𝑓𝑑)

93. Denominator (Invalid-IV)
Derived a closed form
solution.
Properties of dirichlet and
hyperdirichlet distributions.
-Laplace transform
Numerator (Valid-IV)
No closed form solution
exists.
Used Monte Carlo methods
for approximating.
-Annealed Importance
Sampling
93

94. 94

95. 95
Studies from American Economic Review Validity Ratio
Effect of Mexican immigration on crime in United States (2015) 0.07
Effect of subsidy manipulation on Medicare premiums (2015) 1.02
Effect of credit supply on housing prices (2015) 0.01
Effect of Chinese import competition on local labor markets (2013) 0.3
Effect of rural electrification on employment in South Africa (2011) 3.6
Expt: National Job Training Partnership Act (JTPA) Study (2002) 3.4

96. 96

97. 97

98. 98

99. 99
Causal algorithms
Warm Start (choosing expts.)
Online+Offline

100. 100
𝑷 𝑿, 𝒚 : 𝑦 = 𝑘 𝑋 + 𝜖

101. http://www.amitsharma.in
101
1. Hofman, Sharma, and Watts (2017). Prediction and explanation in
social systems. Science, 355.6324.
2. Sharma (2016). Necessary and probably sufficient test for finding
instrumental variables. Working paper.
3. Sharma, Hofman, and Watts (2016). Split-door criterion for causal
identification: An algorithm for finding natural experiments. Under
review at Annals of Applied Statistics (AOAS).
4. Sharma, Hofman, and Watts (2015). Estimating the causal impact of
recommendation systems from observational data. In Proceedings of
the 16th ACM Conference on Economics and Computation.

102. 102

103. 103