An Index-based Deterministic Asymptotically Optimal Algorithm for Constrained Multi-armed Bandit Problems

TL;DR

This paper introduces a deterministic index-based algorithm for constrained multi-armed bandit problems that guarantees asymptotic optimality, with finite-time bounds and extensions to cases where the optimal value is unknown.

Contribution

It presents a novel deterministic algorithm achieving asymptotic optimality in constrained bandits, extending existing methods with finite-time bounds and value estimation.

Findings

The algorithm converges to optimal feasible arms with probability approaching one.

Finite-time bounds are established for the probability of optimality.

Extensions to unknown optimal values are discussed with asymptotic guarantees.

Abstract

For the model of constrained multi-armed bandit, we show that by construction there exists an index-based deterministic asymptotically optimal algorithm. The optimality is achieved by the convergence of the probability of choosing an optimal feasible arm to one over infinite horizon. The algorithm is built upon Locatelli et al.'s "anytime parameter-free thresholding" algorithm under the assumption that the optimal value is known. We provide a finite-time bound to the probability of the asymptotic optimality given as 1-O(|A|Te^{-T}) where T is the horizon size and A is the set of the arms in the bandit. We then study a relaxed-version of the algorithm in a general form that estimates the optimal value and discuss the asymptotic optimality of the algorithm after a sufficiently large T with examples.