Network experimentation at scale

TL;DR

This paper presents a scalable framework for network experimentation at Facebook that accounts for interference between units, improves estimation precision, and demonstrates real-world network effects through case studies.

Contribution

It introduces a cluster-based regression adjustment and logging exposure techniques to better estimate treatment effects in networked online experiments.

Findings

Cluster-based regression improves precision in treatment effect estimation.

Logging exposure reduces variance in experimental estimates.

Real-world case studies reveal significant network effects.

Abstract

We describe our framework, deployed at Facebook, that accounts for interference between experimental units through cluster-randomized experiments. We document this system, including the design and estimation procedures, and detail insights we have gained from the many experiments that have used this system at scale. We introduce a cluster-based regression adjustment that substantially improves precision for estimating global treatment effects as well as testing for interference as part of our estimation procedure. With this regression adjustment, we find that imbalanced clusters can better account for interference than balanced clusters without sacrificing accuracy. In addition, we show how logging exposure to a treatment can be used for additional variance reduction. Interference is a widely acknowledged issue with online field experiments, yet there is less evidence from real-world experiments demonstrating interference in online settings. We fill this gap by describing two case studies that capture significant network effects and highlight the value of this experimentation framework.