Integrating Diagnostic Checks into Estimation

TL;DR

This paper proposes integrating diagnostic checks into estimation by residualizing baseline estimators, which improves inference, reduces variance, and minimizes bias, demonstrated through an RCT application.

Contribution

It introduces a novel residualization method that incorporates diagnostic checks into estimation, enhancing accuracy and robustness in empirical research.

Findings

Residualization increases estimate magnitude and reduces standard error by about 10%.

Method eliminates inference distortions from selective reporting.

Approach reduces variance when the baseline model is correctly specified.

Abstract

Empirical researchers often use diagnostic checks to assess the plausibility of their modeling assumptions, such as testing for covariate balance in RCTs, pre-trends in event studies, or instrument validity in IV designs. While these checks are traditionally treated as external hurdles to estimation, we argue they should be integrated into the estimation process itself. In particular, we propose residualizing one's baseline estimator against the vector of diagnostic check statistics to remove the component of baseline sampling variation explained by the diagnostic checks. This residualized estimator offers researchers a "free lunch," delivering three properties simultaneously: (i) eliminating inference distortions from check-based selective reporting; (ii) reducing variance without changing the estimand when the baseline model is correctly specified; and (iii) minimizing worst-case bias under bounded local misspecification within the class of linear adjustments. We apply our method to the RCT in Kaur et al. (2024) and find that, even in a setting where all balance checks pass comfortably, residualization increases the magnitude of the baseline point estimate and reduces its standard error, equivalent to approximately a 10% increase in sample size.