Asymptotically Optimal Algorithms for Budgeted Multiple Play Bandits

TL;DR

This paper extends multi-armed bandit algorithms to a budgeted setting with multiple plays, deriving regret bounds and proposing asymptotically optimal algorithms based on Thompson sampling and KL-UCB variants.

Contribution

It introduces a budgeted multi-play bandit model, derives regret lower bounds, and demonstrates asymptotic optimality of new algorithms for various reward distributions.

Findings

Derived asymptotic regret lower bounds for the model.

Proposed and analyzed variants of Thompson sampling and KL-UCB.

Proved asymptotic optimality of these algorithms in the budgeted setting.

Abstract

We study a generalization of the multi-armed bandit problem with multiple plays where there is a cost associated with pulling each arm and the agent has a budget at each time that dictates how much she can expect to spend. We derive an asymptotic regret lower bound for any uniformly efficient algorithm in our setting. We then study a variant of Thompson sampling for Bernoulli rewards and a variant of KL-UCB for both single-parameter exponential families and bounded, finitely supported rewards. We show these algorithms are asymptotically optimal, both in rateand leading problem-dependent constants, including in the thick margin setting where multiple arms fall on the decision boundary.