Rising Multi-Armed Bandits with Known Horizons

TL;DR

This paper introduces the Rising Multi-Armed Bandit framework where rewards increase with plays, emphasizing the importance of horizon-aware strategies, and proposes a novel algorithm with theoretical guarantees and empirical success.

Contribution

It presents the first horizon-aware algorithm for Rising Multi-Armed Bandits, with theoretical regret bounds and demonstrated empirical improvements over existing methods.

Findings

The proposed CURE-UCB algorithm outperforms horizon-agnostic strategies.

Theoretical regret bounds are established for the new algorithm.

Empirical results show significant improvements in structured environments.

Abstract

The Rising Multi-Armed Bandit (RMAB) framework models environments where expected rewards of arms increase with plays, which models practical scenarios where performance of each option improves with the repeated usage, such as in robotics and hyperparameter tuning. For instance, in hyperparameter tuning, the validation accuracy of a model configuration (arm) typically increases with each training epoch. A defining characteristic of RMAB is em horizon-dependent optimality: unlike standard settings, the optimal strategy here shifts dramatically depending on the available budget $T$. This implies that knowledge of $T$ yields significantly greater utility in RMAB, empowering the learner to align its decision-making with this shifting optimality. However, the horizon-aware setting remains underexplored. To address this, we propose a novel CUmulative Reward Estimation UCB (CURE-UCB) that explicitly integrates the horizon. We provide a rigorous analysis establishing a new regret upper bound and prove that our method strictly outperforms horizon-agnostic strategies in structured environments like ``linear-then-flat'' instances. Extensive experiments demonstrate its significant superiority over baselines.