An Evaluation of Bounding Approaches for Generalization

TL;DR

This paper evaluates bounding methods as an alternative to propensity score techniques for improving generalization in randomized experiments, especially when assumptions are questionable, by nonparametrically estimating parameters within plausible bounds.

Contribution

It introduces and discusses bounding approaches, including methods to tighten bounds, and demonstrates their application through simulations and a real trial.

Findings

Bounds can be tightened using monotonicity assumptions

Propensity score stratification improves bounds

Simulation studies show conditions for effective bounding

Abstract

Statisticians have recently developed propensity score methods to improve generalizations from randomized experiments that do not employ random sampling. However, these methods typically rely on assumptions whose plausibility may be questionable in practice. In this article, we introduce and discuss bounding, an approach that is based on alternative assumptions that may be more plausible in a given study. The bounding framework nonparametrically estimates population parameters using a range of plausible values that are consistent with the observed characteristics of the data. We illustrate how the bounds can be tightened using three approaches: imposing an alternative assumption based on monotonicity, redefining the population of inference, and using propensity score stratification. Using the results from two simulation studies, we examine the conditions under which bounds for the population parameter are tightened. We conclude with an application of bounding to SimCalc, a cluster randomized trial that evaluated the effectiveness of a technology aid on mathematics achievement.