Assessing the Robustness to an Unobserved Confounder of the Average Treatment Effect on the Treated Estimated by Propensity Score Matching
DOI:
https://doi.org/10.15678/AOC.2016.1504Keywords:
Propensity Score Matching, sensitivity analysis, Rosenbaum’s sensitivity analysis, labour market policyAbstract
One of the serious drawbacks of observational studies is the selection bias caused by the selection process to the treatment group. Propensity Score Matching (PSM), which allows for the reduction of the selection bias when estimating the average treatment effect on the treated (ATT), is a method recommended for the evaluation of projects and programmes co-financed by the European Union. PSM relies on a strong assumption known as the Conditional Independence Assumption (CIA) which implies that selection into the treatment group is based on observable variables, and all variables influencing both the selection process and outcome are observed by the researcher. If this does not hold, the estimated effect may be not so much the result of the treatment as of the lack of balance of an unobserved confounder, which affects both the selection process and the outcome. Rosenbaum’s sensitivity analysis allows researchers to determine how strong the impact of such a potential unobserved confounder on selection into treatment and the outcome must be to undermine conclusions about ATT estimated by PSM. Rosenbaum’s primal and simultaneous approaches are applied in the paper to assess robustness to an unobserved confounder of the net effect of internships for unemployed young people with a maximum age of thirty-five (estimated with PSM) organized by one of the biggest district employment offices in Małopolska.
References
Abadie, A. (2002) “Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models”. Journal of the American Statistical Association 97 (457): 284–92, https://doi.org/10.1198/016214502753479419.
Abadie, A. and Imbens, G. W. (2006) “Large Sample Properties of Matching Estimators for Average Treatment Effects”. Econometrica 74 (1): 235–67, https://doi.org/10.1111/j.1468-0262.2006.00655.x.
Caliendo, M. and Kopeinig, S. (2008) “Some Practical Guidance for the Implementation of Propensity Score Matching”. Journal of Economic Surveys 22 (1): 31–72, https://doi.org/10.1111/j.1467-6419.2007.00527.x.
Denkowska, S. (2015) “Wybrane metody oceny jakości dopasowania w Propensity Score Matching” [Selected quality assessment methods in propensity score matching] in K. Jajuga and M. Walesiak (eds) Taksonomia 24. Klasyfikacja i analiza danych – teoria i zastosowania [Taxonomy 24. Classification and analysis of data – theory and application]. Prace Naukowe Uniwersytetu Ekonomicznego we Wrocławiu No. 384. Wrocław: Wydawnictwo Uniwersytetu Ekonomicznego we Wrocławiu, pp. 60–74.
Denkowska, S. (2016) “Zastosowanie analizy wrażliwości do oceny wpływu nieobserwowanej zmiennej w Propensity Score Matching” [The application of sensitivity analysis in assessing the impact of an unobserved confounder in propensity score matching] in K. Jajuga and M. Walesiak (eds) Taksonomia 27. Klasyfikacja i analiza danych – teoria i zastosowania [Taxonomy 27. Classification and analysis of data – theory and application]. Prace Naukowe Uniwersytetu Ekonomicznego we Wrocławiu No. 427. Wrocław: Wydawnictwo Uniwersytetu Ekonomicznego we Wrocławiu, pp. 66–75.
EC (2014) The Programming Period 2014–2020. Guidance Document on Monitoring and Evaluation. European Regional Development Fund and Cohesion Fund. Concepts and Recommendations, March. Bruxelles: European Commission.
Gastwirth, J., Krieger, A. and Rosenbaum, P. (1998) “Dual and Simultaneous Sensitivity Analysis for Matched Pairs”. Biometrika 85(4): 907–20, https://doi.org/10.1093/biomet/85.4.907.
Heckman, J. J., Ichimura, H., Smith, J. and Todd, P. (1998) “Characterizing Selection Bias Using Experimental Data”. Econometrica 66: 1017–98.
Keele, L. (2010) An Overview of Rebounds: An R Package for Rosenbaum Bounds Sensitivity Analysis with Matched Data, personal.psu.edu/ljk20/rbounds%20vignette.pdf. Accessed: 15 January 2016.
Keele, L. (2014) Package ‘Rbounds’, https://cran.r-project.org/web/packages/rbounds/rbounds.pdf. Accessed: 15 January 2016.
Konarski, R. and Kotnarowski, M. (2007) “Zastosowanie metody propensity score matching w ewaluacji ex-post” [The use of propensity score matching in ex-post evaluations” in A. Huber (ed.) Ewaluacja ex-post. Teoria i praktyka badawcza [Ex-post evaluation. Theory and research practice]. Warszawa: PARP.
Liu, W., Kuramoto, S. K. and Stuart, E. A. (2013) “An Introduction to Sensitivity Analysis for Unobserved Confounding in Non-Experimental Prevention Research”. Prevention Science 14 (6): 570–80, https://doi.org/10.1007/s11121-012-0339-5.
Rosenbaum, P. R. and Rubin, D. B. (1983) “The Central Role of Propensity Score in Observational Studies for Casual Effects”. Biometrika 70 (1): 41–55, https://doi.org/10.1093/biomet/70.1.41.
Rosenbaum, P. R. (2002) Observational Studies. New York: Springer.
Rosenbaum, P. R. (2005) “Observational Study” in B. S. Everitt and D. C. Howell (eds) Encyclopedia of Statistics in Behavioral Science, vol. 3. New York: John Wiley & Sons.
Rosenbaum, P. R. (2010) Design of Observational Studies. New York: Springer.
Rubin, D. (1978) “Bayesian Inference for Causal Effects: The Role of Randomization”. The Annals of Statistics 6 (1): 34–58, https://doi.org/10.1214/aos/1176344064.
Rubin, D. (2001) “Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation”. Health Services & Outcomes Research Methodology 2: 169–88.
Rubin, D. and Thomas, N. (1996) “Matching Using Estimated Propensity Scores: Relating Theory to Practice”. Biometrics 52 (1): 249–64, https://doi.org/10.2307/2533160.
Sekhon, J. S. (2011) “Multivariate and Propensity Score Matching Software with Automated Balance Optimization: The Matching Package for R”. Journal of Statistical Software 42 (7): 1–52, https://doi.org/10.18637/jss.v042.i07.
Smith, J. and Todd, P. (2005) “Does Matching Overcome LaLonde’s Critique of Nonexperimental Estimators?”. Journal of Econometrics 125 (1–2): 305–53, https://doi.org/10.1016/j.jeconom.2004.04.011.
Strawiński, P. (2008) “Quasi-eksperymentalne metody ewaluacji” [Quasi-experimental evaluation methods] in A. Haber (ed.) Środowisko i warsztat ewaluacji [The environment and technique of evaluation]. Warszawa: PARP, pp. 193–220.
Strawiński, P. (2014) Propensity Score Matching. Własności małopróbkowe [Propensity score matching. Small sample properties]. Warszawa: Wydawnictwo Uniwersytetu Warszawskiego.
Stuart, E. A. (2010) “Matching Methods for Causal Inference: a Review and a Look Forward”. Statistical Science 25 (1): 1–21, https://doi.org/10.1214/09-sts313.
Trzciński, R. (2009) Wykorzystanie techniki propensity score matching w badaniach ewaluacyjnych [The use of propensity score matching in evaluation research]. Warszawa: PARP, http://www.parp.gov.pl/index/more/13335. Accessed: December 2014.
Wiśniewski, Z. and Maksim, M. (2013) “Polityka rynku pracy w Polsce – wyniki badań ewaluacyjnych prowadzonych za pomocą metody propensity score matching” [Labour market policy in Poland – the results of evaluation research carried out using propensity score matching] in Rola funduszy unijnych w rozwoju społeczno-gospodarczym regionu [The role of EU funds in the region’s socio-economic development], Zeszyty Naukowe Uniwersytetu Szczecińskiego No. 753. Szczecin: Wydawnictwo Naukowe Uniwersytetu Szczecińskiego, pp. 93–110.
