Bandits with Anytime Knapsacks

TL;DR

This paper introduces a new bandit problem with anytime constraints, proposes an adaptive algorithm called SUAK that balances exploration and exploitation while respecting ongoing cost constraints, and demonstrates its effectiveness through theoretical bounds and simulations.

Contribution

It formulates the bandits with anytime knapsacks problem, proposes the SUAK algorithm, and proves it achieves optimal regret bounds similar to the classic BwK setting.

Findings

SUAK achieves $O(K \, \log T)$ regret bound.

SUAK effectively manages the anytime cost constraint in practice.

Simulations confirm SUAK's practical utility.

Abstract

We consider bandits with anytime knapsacks (BwAK), a novel version of the BwK problem where there is an \textit{anytime} cost constraint instead of a total cost budget. This problem setting introduces additional complexities as it mandates adherence to the constraint throughout the decision-making process. We propose SUAK, an algorithm that utilizes upper confidence bounds to identify the optimal mixture of arms while maintaining a balance between exploration and exploitation. SUAK is an adaptive algorithm that strategically utilizes the available budget in each round in the decision-making process and skips a round when it is possible to violate the anytime cost constraint. In particular, SUAK slightly under-utilizes the available cost budget to reduce the need for skipping rounds. We show that SUAK attains the same problem-dependent regret upper bound of $ O(K \log T)$ established in prior work under the simpler BwK framework. Finally, we provide simulations to verify the utility of SUAK in practical settings.