Quantifying the Internal Validity of Weighted Estimands

TL;DR

This paper develops tools to quantify the internal validity of weighted estimands, such as OLS and 2SLS, by assessing the size of the subpopulation they represent, with applications to policy effect studies.

Contribution

It introduces a framework to evaluate when weighted estimands reflect the average treatment effect for a subpopulation and provides practical diagnostics for internal validity.

Findings

Tools to quantify the subpopulation size underlying weighted estimands.

Application to divorce laws study illustrates diagnostic utility.

Highlights when estimands have low internal validity.

Abstract

In this paper we study a class of weighted estimands, which we define as parameters that can be expressed as weighted averages of the underlying heterogeneous treatment effects. The popular ordinary least squares (OLS), two-stage least squares (2SLS), and two-way fixed effects (TWFE) estimands are all special cases within our framework. Our focus is on answering two questions concerning weighted estimands. First, under what conditions can they be interpreted as the average treatment effect for some (possibly latent) subpopulation? Second, when these conditions are satisfied, what is the upper bound on the size of that subpopulation, either in absolute terms or relative to a target population of interest? We argue that this upper bound provides a valuable diagnostic for empirical research. When a given weighted estimand corresponds to the average treatment effect for a small subset of the population of interest, we say its internal validity is low. Our paper develops practical tools to quantify the internal validity of weighted estimands. We also apply these tools to revisit a prominent study of the effects of unilateral divorce laws on female suicide.