Weight a Minute: Understanding Variability in PATE Estimates Across Target Populations

TL;DR

This paper investigates how the choice of target population affects the bias in estimated treatment effects when using weighting methods like IPSW, emphasizing the importance of selecting representative datasets for valid generalization.

Contribution

It provides empirical evidence on the bias introduced by misaligned target populations in IPSW estimates through simulation studies based on real-world data.

Findings

Bias increases as target populations diverge from the original sample.

Weighting to poorly aligned targets can cause more bias than no weighting.

Selecting an appropriate target population is crucial for valid generalization.

Abstract

Clinical study populations often differ meaningfully from the broader populations to which results are intended to generalize. Weighting methods such as inverse probability of sampling weights (IPSW) reweight study participants to resemble a target population, but the accuracy of these estimates depends heavily on how well the chosen population represents the population of substantive interest. We conduct a simulation study grounded in empirical covariate distributions from several real-world data sources spanning a continuum from highly selective to broadly inclusive populations. Using treatment effect scenarios with varying levels of effect modification, we evaluate IPSW estimators of the population average treatment effect (PATE) across multiple candidate target populations. We quantify the bias that arises when the dataset used to operationalize the target population differs from the intended inference population, even when IPSW is correctly specified. Our results show that bias increases systematically as target populations diverge from a well-representative population, and that weighting to a poorly aligned target can introduce more bias than not weighting at all. These findings highlight that selecting an appropriate target population dataset is a critical design choice for valid generalization.