On Universally Optimal Algorithms for A/B Testing

TL;DR

This paper proves that for two-arm A/B testing in stochastic bandits, no algorithm can outperform uniform sampling universally, and it characterizes the error rates of existing algorithms, revealing limitations and new insights.

Contribution

It establishes the optimality of uniform sampling for two-arm A/B testing and analyzes the asymptotic error rates of the Successive Rejects algorithm for multiple arms.

Findings

No algorithm universally outperforms uniform sampling in two-arm A/B testing.

Uniform sampling matches the lower bound on error rate for two-arm cases.

In multi-arm settings, uniform sampling can outperform Successive Rejects in some instances.

Abstract

We study the problem of best-arm identification with fixed budget in stochastic multi-armed bandits with Bernoulli rewards. For the problem with two arms, also known as the A/B testing problem, we prove that there is no algorithm that (i) performs as well as the algorithm sampling each arm equally (referred to as the {\it uniform sampling} algorithm) in all instances, and that (ii) strictly outperforms uniform sampling on at least one instance. In short, there is no algorithm better than the uniform sampling algorithm. To establish this result, we first introduce the natural class of {\it consistent} and {\it stable} algorithms, and show that any algorithm that performs as well as the uniform sampling algorithm in all instances belongs to this class. The proof then proceeds by deriving a lower bound on the error rate satisfied by any consistent and stable algorithm, and by showing that the uniform sampling algorithm matches this lower bound. Our results provide a solution to the two open problems presented in \citep{qin2022open}. For the general problem with more than two arms, we provide a first set of results. We characterize the asymptotic error rate of the celebrated Successive Rejects (SR) algorithm \citep{audibert2010best} and show that, surprisingly, the uniform sampling algorithm outperforms the SR algorithm in some instances.