A Bracketing Relationship for Long-Term Policy Evaluation with Combined Experimental and Observational Data

TL;DR

This paper develops a bracketing approach to combine experimental and observational data for long-term policy evaluation, providing bounds and insights into the true causal effects under different assumptions.

Contribution

It introduces a novel bracketing relationship that bounds the true effect between LU-based and ECB-based estimands, enhancing robustness in policy evaluation.

Findings

LU-based bounds closely match experimental estimates in education policy.

The sub-martingale property of test scores is key for the bounds.

Sensitivity analysis reveals the role of potential outcomes in the bounds.

Abstract

Combining short-term experimental data with observational data enables credible long-term policy evaluation. The literature offers two key but non-nested assumptions, namely the latent unconfoundedness (LU; Athey et al., 2020) and equi-confounding bias (ECB; Ghassami et al., 2022) conditions, to correct observational selection. Committing to the wrong assumption leads to biased estimation. To mitigate such risks, we provide a novel bracketing relationship (cf. Angrist and Pischke, 2009) repurposed for the setting with data combination: the LU-based estimand and the ECB-based estimand serve as the lower and upper bounds, respectively, with the true causal effect lying in between if either assumption holds. For researchers further seeking point estimates, our Lalonde-style exercise suggests the conservatively more robust LU-based lower bounds align closely with the hold-out experimental estimates for educational policy evaluation. We investigate the economic substantives of these findings through the lens of a nonparametric class of selection mechanisms and sensitivity analysis. We uncover as key the sub-martingale property and sufficient-statistics role (Chetty, 2009) of the potential outcomes of student test scores (Chetty et al., 2011, 2014).