Diaz, Juan Jose, and Sudhanshu Handa. 2006. “An Assessment of Propensity Score Matching as a Nonexperimental Impact Estimator: Evidence from Mexico’s PROGRESA Program.” Journal of Human Resources 41(2): 319–345.
Not all policy questions can be addressed by social experiments. Nonexperimental evaluation methods provide an alternative to experimental designs but their results depend on untestable assumptions. This paper presents evidence on the reliability of propensity score matching (PSM), which estimates treatment effects under the assumption of selection on observables, using a social experiment designed to evaluate the PROGRESA program in Mexico. We find that PSM performs well for outcomes that are measured comparably across survey instruments and when a rich set of control variables is available. However, even small differences in the way outcomes are measured can lead to bias in the technique.
Juan Jose Diaz is an economist at the Group for the Analysis of Development (GRADE) in Lima, Peru. Sudhanshu Handa is associate professor in the Department of Public Policy at the University of North Carolina at Chapel Hill and fellow at the Carolina Population Center. This paper could not have been written without the assistance of Monica Orozco of PROGRESA, who provided essential data and explained operational details of the program to us. The authors thank Jeff Smith and an anonymous reviewer for constructive comments on earlier versions of this manuscript. The data used in this paper are available from Progresa in Mexico City. Other scholars wishing for guidance about how to apply for the data may contact Handa at shanda@email.unc.edu.