Adaptive Algorithms for Infinitely Many-Armed Bandits: A Unified Framework

TL;DR

This paper introduces a distribution-free algorithm, OSE, for infinitely many-armed bandit problems with limited budgets, focusing on maximizing expected simple reward and providing theoretical guarantees across various distribution classes.

Contribution

The paper proposes the OSE algorithm that adapts to distributional properties and achieves near-optimal rates, along with the practical PROSE variant for improved empirical performance.

Findings

OSE achieves near-optimal sample complexity across distribution classes.

Transition regimes depend on the noise level and distribution parameters.

PROSE outperforms existing methods empirically in main distribution classes.

Abstract

We consider a bandit problem where the buget is smaller than the number of arms, which may be infinite. In this regime, the usual objective in the literature is to minimize simple regret. To analyze broad classes of distributions with potentially unbounded support, where simple regret may not be well-defined, we take a slightly different approach and seek to maximize the expected simple reward of the recommended arm, providing anytime guarantees. To that end, we introduce a distribution-free algorithm, OSE, that adapts to the distribution of arm means and achieves near-optimal rates for several distribution classes. We characterize the sample complexity through the rank-corrected inverse squared gap function. In particular, we recover known upper bounds and transition regimes for $\alpha$ less or greater than $1/2$ when the quantile function is $\lambda_\eta = 1-\eta^{\alpha}$. We additionally identify new transition regimes depending on the noise level relative to $\alpha$, which we conjecture to be nearly optimal. Additionally, we introduce an enhanced practical version, PROSE, that achieves state-of-the-art empirical performance for the main distribution classes considered in the literature.