On the probability of invalidating a causal inference due to limited external validity

TL;DR

This paper introduces a probabilistic measure called PEV to assess how limited external validity can invalidate causal inferences in empirical research, linking it to statistical power and providing a practical evaluation guideline.

Contribution

It conceptualizes the gap between observed and ideal samples as unobserved sample and quantifies its impact on causal inference validity using PEV, with an empirical example.

Findings

PEV quantifies the risk of invalidating causal inference due to external validity issues.

PEV is related to statistical power in NHST.

Guidelines for evaluating external validity using PEV are provided.

Abstract

External validity is often questionable in empirical research, especially in randomized experiments due to the trade-off between internal validity and external validity. To quantify the robustness of external validity, one must first conceptualize the gap between a sample that is fully representative of the target population (i.e., the ideal sample) and the observed sample. Drawing on Frank & Min (2007) and Frank et al. (2013), I define such gap as the unobserved sample and intend to quantify its relationship with the null hypothesis statistical testing (NHST) in this study. The probability of invalidating a causal inference due to limited external validity, i.e., the PEV, is the probability of failing to reject the null hypothesis based on the ideal sample provided the null hypothesis has been rejected based on the observed sample. This study illustrates the guideline and the procedure of evaluating external validity with the PEV through an empirical example (i.e., Borman et al. (2008)). Specifically, one would be able to locate the threshold of the unobserved sample statistic that would make the PEV higher than a desired value and use this information to characterize the unobserved sample that would render external validity of the research in question less robust. The PEV is shown to be linked to statistical power when the NHST is thought to be based on the ideal sample.