Randomization Inference for Treatment Effect Variation

TL;DR

This paper introduces a model-free, randomization-based method to test for unexplained treatment effect variation in randomized experiments, addressing the nuisance of average treatment effects and extending to heterogeneity testing.

Contribution

It proposes a finite-sample valid testing approach for treatment effect heterogeneity that does not rely on parametric models, applicable to large-scale randomized studies.

Findings

Significant unexplained treatment effect variation found in the Head Start Impact Study.

Method guarantees valid tests even with small sample sizes.

Extension to testing heterogeneity beyond specified models.

Abstract

Applied researchers are increasingly interested in whether and how treatment effects vary in randomized evaluations, especially variation not explained by observed covariates. We propose a model-free approach for testing for the presence of such unexplained variation. To use this randomization-based approach, we must address the fact that the average treatment effect, generally the object of interest in randomized experiments, actually acts as a nuisance parameter in this setting. We explore potential solutions and advocate for a method that guarantees valid tests in finite samples despite this nuisance. We also show how this method readily extends to testing for heterogeneity beyond a given model, which can be useful for assessing the sufficiency of a given scientific theory. We finally apply our method to the National Head Start Impact Study, a large-scale randomized evaluation of a Federal preschool program, finding that there is indeed significant unexplained treatment effect variation.