Unified theory of upper confidence bound policies for bandit problems targeting total reward, maximal reward, and more

TL;DR

This paper unifies the analysis of UCB policies for total reward and max bandit problems, introducing the oracle quantity concept and proposing new algorithms with proven order optimality and practical effectiveness.

Contribution

It provides a unified theoretical framework for UCB policies across different bandit problems, introduces the oracle quantity concept, and proposes PIUCB algorithms for improved performance.

Findings

MaxSearch algorithm is order-optimal for max bandit problem.

Confidence intervals of the oracle quantity are crucial for UCB optimality.

PIUCB algorithms perform comparably or better than MaxSearch in experiments.

Abstract

The upper confidence bound (UCB) policy is recognized as an order-optimal solution for the classical total-reward bandit problem. While similar UCB-based approaches have been applied to the max bandit problem, which aims to maximize the cumulative maximal reward, their order optimality remains unclear. In this study, we clarify the unified conditions under which the UCB policy achieves the order optimality in both total-reward and max bandit problems. A key concept of our theory is the oracle quantity, which identifies the best arm by its highest value. This allows a unified definition of the UCB policy as pulling the arm with the highest UCB of the oracle quantity. Additionally, under this setting, optimality analysis can be conducted by replacing traditional regret with the number of failures as a core measure. One consequence of our analysis is that the confidence interval of the oracle quantity must narrow appropriately as trials increase to ensure the order optimality of UCB policies. From this consequence, we prove that the previously proposed MaxSearch algorithm satisfies this condition and is an order-optimal policy for the max bandit problem. We also demonstrate that new bandit problems and their order-optimal UCB algorithms can be systematically derived by providing the appropriate oracle quantity and its confidence interval. Building on this, we propose PIUCB algorithms, which aim to pull the arm with the highest probability of improvement (PI). These algorithms can be applied to the max bandit problem in practice and perform comparably or better than the MaxSearch algorithm in toy examples. This suggests that our theory has the potential to generate new policies tailored to specific oracle quantities.