Experimentation for Homogenous Policy Change

TL;DR

This paper explores experimental designs for homogeneous policy changes under interference, showing that difference-in-means estimators and two-stage randomization can outperform regression models and one-stage designs in finite samples, with applications to Facebook experiments.

Contribution

It introduces methods for experimental design under interference, demonstrating the advantages of difference-in-means and two-stage randomization for estimating global average treatment effects.

Findings

Difference-in-means estimators outperform regression models in finite samples.

Two-stage randomization with intra-cluster correlation improves estimation.

Simulations and Facebook experiments validate the proposed methods.

Abstract

When the Stable Unit Treatment Value Assumption is violated and there is interference among units, there is not a uniquely defined Average Treatment Effect, and alternative estimands may be of interest. Among these are average unit-level differences in outcomes under different homogeneous treatment policies. We refer to such targets as Global Average Treatment Effects. We consider approaches to experimental design with multiple treatment conditions under partial interference and, given the estimand of interest, we show that difference-in-means estimators may perform better than correctly specified regression models in finite samples on root mean squared error for such targets. With errors correlated at the cluster level, we demonstrate that two-stage randomization procedures with intra-cluster correlation of treatment strictly between zero and one may dominate one-stage randomization designs on the same metric. Simulations illustrate performance of this approach; we consider an application to online experiments at Facebook.