Optimism in the Face of Ambiguity Principle for Multi-Armed Bandits

TL;DR

This paper introduces a novel FTPL algorithm for multi-armed bandits that balances computational efficiency with optimal regret, using ambiguity-aware perturbations and a new arm sampling method.

Contribution

It proposes an ambiguity-based FTPL algorithm that unifies and generalizes existing methods, achieving optimal regret with low computational costs.

Findings

Achieves optimal regret for adversarial and stochastic bandits.

Provides a bisection algorithm up to 10,000 times faster than standard methods.

Generalizes existing FTPL and FTRL algorithms, settling conjectures.

Abstract

Follow-The-Regularized-Leader (FTRL) algorithms often enjoy optimal regret for adversarial as well as stochastic bandit problems and allow for a streamlined analysis. Nonetheless, FTRL algorithms require the solution of an optimization problem in every iteration and are thus computationally challenging. In contrast, Follow-The-Perturbed-Leader (FTPL) algorithms achieve computational efficiency by perturbing the estimates of the rewards of the arms, but their regret analysis is cumbersome. We propose a new FTPL algorithm that generates optimal policies for both adversarial and stochastic multi-armed bandits. Like FTRL, our algorithm admits a unified regret analysis, and similar to FTPL, it offers low computational costs. Unlike existing FTPL algorithms that rely on independent additive disturbances governed by a \textit{known} distribution, we allow for disturbances governed by an \textit{ambiguous} distribution that is only known to belong to a given set and propose a principle of optimism in the face of ambiguity. Consequently, our framework generalizes existing FTPL algorithms. It also encapsulates a broad range of FTRL methods as special cases, including several optimal ones, which appears to be impossible with current FTPL methods. Finally, we use techniques from discrete choice theory to devise an efficient bisection algorithm for computing the optimistic arm sampling probabilities. This algorithm is up to $10^4$ times faster than standard FTRL algorithms that solve an optimization problem in every iteration. Our results not only settle existing conjectures but also provide new insights into the impact of perturbations by mapping FTRL to FTPL.