Designing Randomized Experiments to Predict Unit-Specific Treatment Effects

TL;DR

This paper advocates designing randomized experiments specifically to predict individual treatment effects within a population, analyzing how sampling and modeling choices influence prediction accuracy and bias.

Contribution

It introduces a framework for designing experiments aimed at accurate unit-specific treatment effect prediction, emphasizing the impact of sampling and model selection.

Findings

Sampling differences can significantly bias predictions.

Model choice affects bias, variance, and mean squared error.

Average treatment effect estimates may outperform unit-specific models in some cases.

Abstract

Typically, a randomized experiment is designed to test a hypothesis about the average treatment effect and sometimes hypotheses about treatment effect variation. The results of such a study may then be used to inform policy and practice for units not in the study. In this paper, we argue that given this use, randomized experiments should instead be designed to predict unit-specific treatment effects in a well-defined population. We then consider how different sampling processes and models affect the bias, variance, and mean squared prediction error of these predictions. The results indicate, for example, that problems of generalizability (differences between samples and populations) can greatly affect bias both in predictive models and in measures of error in these models. We also examine when the average treatment effect estimate outperforms unit-specific treatment effect predictive models and implications of this for planning studies.