Estimating treatment-effect heterogeneity across sites, in multi-site randomized experiments with few units per site

TL;DR

This paper develops methods to estimate and analyze heterogeneity of treatment effects across sites in multi-site randomized trials with limited units per site, accounting for imperfect compliance and unobserved site characteristics.

Contribution

It introduces new estimators for treatment-effect heterogeneity and site-level effects, addressing challenges in multi-site experiments with few units per site and imperfect compliance.

Findings

Significant treatment-effect heterogeneity found across sites.

Negative correlation between treatment effects and sites' baseline outcomes.

Methodology applied to real data from French job placement agencies.

Abstract

In multi-site randomized trials with many sites and few randomization units per site, an Empirical-Bayes estimator can be used to estimate the variance of the treatment effect across sites. When this estimator indicates that treatment effects do vary, we propose estimators of the coefficients from regressions of site-level effects on site-level characteristics that are unobserved but can be unbiasedly estimated, such as sites' average outcome without treatment, or site-specific treatment effects on mediator variables. In experiments with imperfect compliance, we show that the sign of the correlation between local average treatment effects (LATEs) and site-level characteristics is identified, and we propose a partly testable assumption under which the variance of LATEs is identified. We use our results to revisit Behaghel et al (2014), who study the effect of counseling programs on job seekers' job-finding rate, in 200 job placement agencies in France. We find considerable treatment-effect heterogeneity, both for intention to treat and LATE effects, and the treatment effect is negatively correlated with sites' job-finding rate without treatment.