Query-Reward Tradeoffs in Multi-Armed Bandits

TL;DR

This paper studies a multi-armed bandit setting where rewards are only observed upon query, providing tight bounds on regret and queries, and introduces a new adaptive UCB-style algorithm that handles multiple optimal arms effectively.

Contribution

It introduces a novel UCB-style sampling method that adapts to the number of optimal arms and establishes tight bounds for regret and query complexity in this setting.

Findings

Fundamental difference between problems with single and multiple optimal arms.

New UCB-style algorithm adapts to the number of optimal arms.

Tight problem-dependent bounds on regret and queries.

Abstract

We consider a stochastic multi-armed bandit setting where reward must be actively queried for it to be observed. We provide tight lower and upper problem-dependent guarantees on both the regret and the number of queries. Interestingly, we prove that there is a fundamental difference between problems with a unique and multiple optimal arms, unlike in the standard multi-armed bandit problem. We also present a new, simple, UCB-style sampling concept, and show that it naturally adapts to the number of optimal arms and achieves tight regret and querying bounds.