Design Experiments to Compare Multi-armed Bandit Algorithms

TL;DR

The paper introduces Artificial Replay, a new experimental design for comparing multi-armed bandit algorithms that reduces data requirements and variance, enabling more efficient and reliable policy evaluation.

Contribution

It proposes a novel Artificial Replay method that reuses recorded trajectories to compare bandit algorithms, reducing experimental cost and variance while maintaining unbiasedness.

Findings

Artificial Replay halves the data needed for reliable comparison.

The estimator's variance grows sub-linearly with the number of users.

Numerical experiments confirm theoretical advantages with popular bandit algorithms.

Abstract

Online platforms routinely compare multi-armed bandit algorithms, such as UCB and Thompson Sampling, to select the best-performing policy. Unlike standard A/B tests for static treatments, each run of a bandit algorithm over $T$ users produces only one trajectory, because the algorithm's decisions depend on all past interactions. Reliable inference therefore demands many independent restarts of the algorithm, making experimentation costly and delaying deployment decisions. We propose Artificial Replay (AR) as a new experimental design for this problem. AR first runs one policy and records its trajectory. When the second policy is executed, it reuses a recorded reward whenever it selects an action the first policy already took, and queries the real environment only otherwise. We develop a new analytical framework for this design and prove three key properties of the resulting estimator: it is unbiased; it requires only $T + o(T)$ user interactions instead of $2T$ for a run of the treatment and control policies, nearly halving the experimental cost when both policies have sub-linear regret; and its variance grows sub-linearly in $T$, whereas the estimator from a na\"ive design has a linearly-growing variance. Numerical experiments with UCB, Thompson Sampling, and $\epsilon$-greedy policies confirm these theoretical gains.