Algorithm Design and Stronger Guarantees for the Improving Multi-Armed Bandits Problem

TL;DR

This paper introduces new algorithms for the improving multi-armed bandits problem, providing stronger theoretical guarantees and empirical performance, especially under certain reward curve conditions.

Contribution

It proposes two parameterized families of algorithms with improved sample complexity bounds and guarantees, extending prior work with stronger theoretical and empirical results.

Findings

Achieved stronger guarantees with optimal dependence on the number of arms.

Bounded sample complexity for learning near-optimal algorithms from offline data.

Empirical evaluations demonstrated effectiveness on hyperparameter tuning benchmarks.

Abstract

The improving multi-armed bandits problem is a formal model for allocating effort under uncertainty, motivated by scenarios such as investing research effort into new technologies, performing clinical trials, and hyperparameter selection from learning curves. Each pull of an arm provides reward that increases monotonically with diminishing returns. A growing line of work has designed algorithms for improving bandits, albeit with somewhat pessimistic worst-case guarantees. Indeed, strong lower bounds of $\Omega(k)$ and $\Omega(\sqrt{k})$ multiplicative approximation factors are known for both deterministic and randomized algorithms (respectively) relative to the optimal arm, where $k$ is the number of bandit arms. In this work, we propose two new parameterized families of bandit algorithms and bound the sample complexity of learning the near-optimal algorithm from each family using offline data. We also perform empirical evaluations on standard hyperparameter tuning benchmarks. The first family we define includes the optimal randomized algorithm from prior work. We show that an appropriately chosen algorithm from this family can achieve stronger guarantees, with optimal dependence on $k$, when the arm reward curves satisfy additional properties related to the strength of concavity. Our second family contains algorithms that both guarantee best-arm identification on well-behaved instances and revert to worst-case guarantees on poorly-behaved instances.