Randomization and Its Discontents

1. Peter M. Lance, PhD MEASURE Evaluation University of North Carolina at Chapel Hill JUNE 29, 2016 Randomization and Its Discontents

2. Global, five-year, $180M cooperative agreement Strategic objective: To strengthen health information systems – the capacity to gather, interpret, and use data – so countries can make better decisions and sustain good health outcomes over time. Project overview

3. Improved country capacity to manage health information systems, resources, and staff Strengthened collection, analysis, and use of routine health data Methods, tools, and approaches improved and applied to address health information challenges and gaps Increased capacity for rigorous evaluation Phase IV Results Framework

4. Global footprint (more than 25 countries)

5. • The program impact evaluation challenge • Randomization • Selection on observables • Within estimators • Instrumental variables

6. • The program impact evaluation challenge • Randomization • Selection on observables • Within estimators • Instrumental variables

9. Did the program make a difference?

10. Did the program cause a change in an outcome of interest Y ? (Causality)

11. What happens if the individual participates {Causal} Program impact 𝑌𝑖 1 − 𝑌𝑖 0 = Program impact What happens if the individual does not participate

12. Average treatment effect (ATE) 𝐸 𝑌1 − 𝑌0 Average effect of treatment on the treated (ATT) 𝐸 𝑌1 − 𝑌0|𝑃 = 1 Treatment effects

13. 𝑌𝑖 1 , 𝑌𝑖 0 Observed outcome

17. 𝑌𝑖 1 , 𝑌𝑖 0 Observed outcome Fundamental Identification Problem of Program Impact Evaluation

18. 𝑌𝑖 1 , 𝑌𝑖 0 Observed outcome Fundamental identification problem of program impact evaluation

19. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1

20. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1

21. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1

25. Impact evaluation 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1

28. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1 Average Y across sample of participants − Average Y Across Sample of Non−Participants

29. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1 Average Y across sample of participants − Average Y across sample of non−participants

34. Impact evaluation 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1 𝐸 𝑌0|𝑃 = 0 = 𝐸(𝑌0|𝑃 = 1)

35. 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 𝐸 𝑌0|𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 1

36. 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 𝐸 𝑌0|𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 1

37. X Y P

38. X Y P 𝑌 = 𝑃 ∙ 𝑌1 + 1 − 𝑃 ∙ 𝑌0

39. X Y P

40. X Y P

42. The big idea (part un) So, the basic idea of the randomization/experimental approach to impact evaluation is that by randomizing program participation we insure that participants and non-participants are alike on average in terms of their characteristics

43. The big idea (part deux) If this is the case, then any differences in average outcomes between the two groups can be ascribed to the one way in which they do differ: program participation

44. 1.Participants in the experiment are randomly selected from the population of interest and randomly assigned to their program participation status; 2.All participants in the trial comply with the program participation status to which they are assigned; 3.The experiment lasts long enough to replicate the program under consideration and to influence outcomes; 4.There are no social interactions that may make a full-scale program inherently different from a smaller scale intervention

46. Source: Washington Policy Center

47. Source: EH.net

48. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 𝑌 = 𝑃 ∙ 𝑌1 + 1 − 𝑃 ∙ 𝑌0

49. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 Average Y Across Sample of Participants − Average Y Across Sample of Non−Participants

50. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 Average Y across sample of participants − Average Y Across Sample of Non−Participants

51. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 Average Y across sample of participants − Average Y across sample of non−participants

52. X Y P

53. 𝐸 𝑋|𝑃 = 1 ≠ 𝐸 𝑋|𝑃 = 0

54. X Y P

55. 𝐸 𝑌1|𝑃 = 1 > 𝐸 𝑌1 > 𝐸 𝑌1|𝑃 = 0

56. 𝐸 𝑌0|𝑃 = 0 < 𝐸 𝑌0 < 𝐸 𝑌0|𝑃 = 1

57. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 (Overestimated) (Underestimated)

60. X Y P

61. X Y P

64. Refusal rates by plan Plan Refusal rate (%) Free 8 25 and 50% coinsurance 11 95% coinsurance 25 Source: Newhouse et al. (1993)

73. X Y P

74. X Y P

75. X Y P

76. If such and such is true of the real world processes that gave rise to program participation and outcomes in our observed non-experimental sample, then the estimates of program impact generated by this quasi- experimental estimator provide the causal impact of program participation on outcomes of interest.

77. Lalonde’s critique of non- experimental estimators

78. Lalonde’s critique of non- experimental estimators

80. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs carried out in a limited or pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward

81. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs in a limited, pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward

86. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0

87. 𝑚𝑒𝑑𝑖𝑎𝑛 𝑌1 − 𝑌0 ≠ 𝑚𝑒𝑑𝑖𝑎𝑛 𝑌1 − 𝑚𝑒𝑑𝑖𝑎𝑛 𝑌0

91. 1.Participants in the experiment are randomly selected from the population of interest and randomly assigned to their program participation status; 2.All participants in the trial comply with the program participation status to which they are assigned; 3.The experiment lasts long enough to replicate the program under consideration and to influence outcomes; 4.There are no social interactions that may make a full-scale program inherently different from a smaller scale intervention

92. Time Approval/ acceptance

93. Time Approval/ acceptance Phase 1

103. Conclusion

104. Conclusion

105. Conclusion

106. Conclusion

107. Links: The manual: http://www.measureevaluation.org/resources/publications/ms- 14-87-en The webinar introducing the manual: http://www.measureevaluation.org/resources/webinars/metho ds-for-program-impact-evaluation My email: pmlance@email.unc.edu

108. MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) under terms of Cooperative Agreement AID-OAA-L-14-00004 and implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International, John Snow, Inc., Management Sciences for Health, Palladium Group, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. www.measureevaluation.org

Randomization and Its Discontents

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Randomization and Its Discontents

Similar to Randomization and Its Discontents (20)

More from MEASURE Evaluation

More from MEASURE Evaluation (20)

Recently uploaded

Recently uploaded (20)

Randomization and Its Discontents