The probability of a robust inference for internal validity and its applications in regression models

TL;DR

This paper introduces the probability of a robust inference (PIV) for assessing internal validity in regression models, formalizing its relationship with null hypothesis testing and providing a six-step evaluation procedure with empirical illustration.

Contribution

It formalizes the PIV concept for internal validity, linking it with NHST and offers a novel six-step process for evaluation in observational studies.

Findings

PIV quantifies the likelihood of consistent null hypothesis rejection.

The method accounts for unobserved samples and bounded beliefs about counterfactuals.

Empirical example demonstrates practical application of the procedure.

Abstract

The internal validity of observational study is often subject to debate. In this study, we define the unobserved sample based on the counterfactuals and formalize its relationship with the null hypothesis statistical testing (NHST) for regression models. The probability of a robust inference for internal validity, i.e., the PIV, is the probability of rejecting the null hypothesis again based on the ideal sample which is defined as the combination of the observed and unobserved samples, provided the same null hypothesis has already been rejected for the observed sample. When the unconfoundedness assumption is dubious, one can bound the PIV of an inference based on bounded belief about the mean counterfactual outcomes, which is often needed in this case. Essentially, the PIV is statistical power of the NHST that is thought to be built on the ideal sample. We summarize the process of evaluating internal validity with the PIV into a six-step procedure and illustrate it with an empirical example (i.e., Hong and Raudenbush (2005)).