The power of A/B testing under interference

TL;DR

This paper develops a theoretical framework to assess and estimate the power of A/B tests under interference in networked environments, enabling better test design and interpretation.

Contribution

It introduces a method to quantify and estimate A/B test power considering interference effects using network models and central limit theorem derivations.

Findings

Interference impacts A/B test sensitivity and power.

A central limit theorem for exposed individuals under Bernoulli interference.

Application of the method to Facebook and Twitter networks.

Abstract

In this paper, we address the fundamental statistical question: how can you assess the power of an A/B test when the units in the study are exposed to interference? This question is germane to many scientific and industrial practitioners that rely on A/B testing in environments where control over interference is limited. We begin by proving that interference has a measurable effect on its sensitivity, or power. We quantify the power of an A/B test of equality of means as a function of the number of exposed individuals under any interference mechanism. We further derive a central limit theorem for the number of exposed individuals under a simple Bernoulli switching interference mechanism. Based on these results, we develop a strategy to estimate the power of an A/B test when actors experience interference according to an observed network model. We demonstrate how to leverage this theory to estimate the power of an A/B test on units sharing any network relationship, and highlight the utility of our method on two applications - a Facebook friendship network as well as a large Twitter follower network. These results yield, for the first time, the capacity to understand how to design an A/B test to detect, with a specified confidence, a fixed measurable treatment effect when the A/B test is conducted under interference driven by networks.