Adaptive Neyman Allocation

TL;DR

This paper introduces an adaptive Neyman allocation method for multi-stage experiments that optimally allocates units to treatment and control groups based on estimated standard deviations, improving statistical power.

Contribution

It proposes a simple adaptive algorithm for Neyman allocation that nearly achieves the theoretical optimal performance in multi-stage experimental settings.

Findings

Algorithm nearly matches the information-theoretic limit

Effective in online A/B testing scenarios

Provides theoretical guarantees for estimation and inference

Abstract

In the experimental design literature, Neyman allocation refers to the practice of allocating units into treated and control groups, potentially in unequal numbers proportional to their respective standard deviations, with the objective of minimizing the variance of the treatment effect estimator. This widely recognized approach increases statistical power in scenarios where the treated and control groups have different standard deviations, as is often the case in social experiments, clinical trials, marketing research, and online A/B testing. However, Neyman allocation cannot be implemented unless the standard deviations are known in advance. Fortunately, the multi-stage nature of the aforementioned applications allows the use of earlier stage observations to estimate the standard deviations, which further guide allocation decisions in later stages. In this paper, we introduce a competitive analysis framework to study this multi-stage experimental design problem. We propose a simple adaptive Neyman allocation algorithm, which almost matches the information-theoretic limit of conducting experiments. We provide theory for estimation and inference using data collected from our adaptive Neyman allocation algorithm. We demonstrate the effectiveness of our adaptive Neyman allocation algorithm using both online A/B testing data from a social media site and synthetic data.