Is it who you are or where you are? Accounting for compositional differences in cross-site treatment variation

TL;DR

This paper introduces a method to distinguish between compositional and contextual treatment effect variations in multisite trials by using covariate balancing weights, enabling better understanding of site-specific effects.

Contribution

The paper develops a novel framework for effect transportation in multisite trials using approximate balancing weights to isolate contextual effects.

Findings

Effective covariate balancing achieved in simulations

Method successfully applied to welfare-to-work trial data

Available implementation in the balancer R package

Abstract

Multisite trials, in which treatment is randomized separately in multiple sites, offer a unique opportunity to disentangle treatment effect variation due to "compositional" differences in the distributions of unit-level features from variation due to "contextual" differences in site-level features. In particular, if we can re-weight (or "transport") each site to have a common distribution of unit-level covariates, the remaining effect variation captures contextual differences across sites. In this paper, we develop a framework for transporting effects in multisite trials using approximate balancing weights, where the weights are chosen to directly optimize unit-level covariate balance between each site and the target distribution. We first develop our approach for the general setting of transporting the effect of a single-site trial. We then extend our method to multisite trials, assess its performance via simulation, and use it to analyze a series of multisite trials of welfare-to-work programs. Our method is available in the balancer R package.