Reservoir Designs for Online Paired Experiments

TL;DR

This paper introduces a reservoir-based pairing method for online experiments that improves variance reduction by effectively matching units with similar covariates in real-time, addressing the challenge of immediate treatment assignment.

Contribution

It proposes a novel reservoir design for online paired experiments that achieves variance improvements without prior covariate knowledge, unlike previous methods.

Findings

Reservoir design improves variance reduction in online experiments.

The method outperforms IID sampling and previous pairing strategies.

Theoretical conditions for asymptotic variance improvement are established.

Abstract

We study the question of how best to stratify units into matched pairs in online experiments, so that units within a pair receive opposite treatment. Past work by Bai, Romano, and Shaikh (2022) has demonstrated the asymptotic variance improvement that comes from pairing units with similar covariates in this way. However, their method requires knowing the covariates for all units a priori; this is not the case in many A/B testing problems, in which units arrive one at a time and must have treatment assigned immediately. Inspired by the terminology of Kapelner and Krieger (2014), we thus introduce the notion of a reservoir design, which maintains a reservoir of unpaired units that can potentially be paired with an incoming unit. We construct a particular reservoir design that uses a distance-based criterion to determine pairing and, via a packing argument, prove conditions under which it attains the asymptotic variance improvement of Bai, Romano, and Shaikh (2022). We illustrate our reservoir design on synthetic and semi-synthetic examples and find improved performance relative to both IID sampling and the design of Kapelner and Krieger (2014).