Bandit Allocational Instability

TL;DR

This paper introduces a new metric called allocation variability for multi-armed bandit algorithms, revealing a fundamental trade-off with regret and providing bounds and an algorithm that achieve near-optimal performance.

Contribution

It establishes a fundamental trade-off between allocation variability and regret in bandit algorithms, introduces a tunable algorithm UCB-f, and resolves an open question in the field.

Findings

Worst-case regret and allocation variability satisfy R_T * S_T=Ω(T^{3/2})

Any sublinear regret algorithm must have S_T=ω(√T)

UCB-f achieves the Pareto optimal trade-off

Abstract

When multi-armed bandit (MAB) algorithms allocate pulls among competing arms, the resulting allocation can exhibit huge variation. This is particularly harmful in modern applications such as learning-enhanced platform operations and post-bandit statistical inference. Thus motivated, we introduce a new performance metric of MAB algorithms termed allocation variability, which is the largest (over arms) standard deviation of an arm's number of pulls. We establish a fundamental trade-off between allocation variability and regret, the canonical performance metric of reward maximization. In particular, for any algorithm, the worst-case regret $R_T$ and worst-case allocation variability $S_T$ must satisfy $R_T \cdot S_T=\Omega(T^{\frac{3}{2}})$ as $T\rightarrow\infty$, as long as $R_T=o(T)$. This indicates that any minimax regret-optimal algorithm must incur worst-case allocation variability $\Theta(T)$, the largest possible scale; while any algorithm with sublinear worst-case regret must necessarily incur ${S}_T= \omega(\sqrt{T})$. We further show that this lower bound is essentially tight, and that any point on the Pareto frontier $R_T \cdot S_T=\tilde{\Theta}(T^{3/2})$ can be achieved by a simple tunable algorithm UCB-f, a generalization of the classic UCB1. Finally, we discuss implications for platform operations and for statistical inference, when bandit algorithms are used. As a byproduct of our result, we resolve an open question of Praharaj and Khamaru (2025).