On the minimum strength of (unobserved) covariates to overturn an insignificant result

TL;DR

This paper analyzes the conditions under which adding unobserved covariates can change a regression result from insignificant to significant, providing a framework for sensitivity analysis and understanding reversal patterns in empirical studies.

Contribution

It characterizes the minimum association strength needed for covariates to overturn an insignificant result, highlighting the difficulty of reversing significance solely by reducing standard error.

Findings

Overturning insignificance requires covariates to alter the point estimate, not just reduce standard error.

The paper provides bounds on maximum t-values given different covariate subsets.

Results explain empirical reversal patterns, such as those documented by Lenz and Sahn (2021).

Abstract

We study conditions under which the addition of variables to a regression equation can turn a previously statistically insignificant result into a significant one. Specifically, we characterize the minimum strength of association required for these variables--both with the dependent and independent variables, or with the dependent variable alone--to elevate the observed t-statistic above a specified significance threshold. Interestingly, we show that it is considerably difficult to overturn a statistically insignificant result solely by reducing the standard error. Instead, included variables must also alter the point estimate to achieve such reversals in practice. Our results can be used to conduct sensitivity analyses against unobserved variables and to bound the maximum t-value one can obtain given different subsets of observed covariates, and may also offer algebraic explanations for patterns of reversals seen in empirical research, such as those documented by Lenz and Sahn (2021).