Tracking Most Significant Shifts in Infinite-Armed Bandits

TL;DR

This paper develops a parameter-free, optimal regret algorithm for infinite-armed bandits with non-stationary rewards, introducing a blackbox scheme and a notion of significant shifts to adaptively improve performance.

Contribution

It presents the first parameter-free, optimal regret bounds for all regimes of non-stationarity in infinite-armed bandits, relaxing distributional assumptions and introducing a shift-based analysis.

Findings

Achieves parameter-free optimal regret bounds for all non-stationarity regimes.

Introduces a blackbox scheme converting finite-armed algorithms for non-stationary environments.

Shows that rising rewards do not impact the difficulty of non-stationarity.

Abstract

We study an infinite-armed bandit problem where actions' mean rewards are initially sampled from a reservoir distribution. Most prior works in this setting focused on stationary rewards (Berry et al., 1997; Wang et al., 2008; Bonald and Proutiere, 2013; Carpentier and Valko, 2015) with the more challenging adversarial/non-stationary variant only recently studied in the context of rotting/decreasing rewards (Kim et al., 2022; 2024). Furthermore, optimal regret upper bounds were only achieved using parameter knowledge of non-stationarity and only known for certain regimes of regularity of the reservoir. This work shows the first parameter-free optimal regret bounds for all regimes while also relaxing distributional assumptions on the reservoir. We first introduce a blackbox scheme to convert a finite-armed MAB algorithm designed for near-stationary environments into a parameter-free algorithm for the infinite-armed non-stationary problem with optimal regret guarantees. We next study a natural notion of significant shift for this problem inspired by recent developments in finite-armed MAB (Suk & Kpotufe, 2022). We show that tighter regret bounds in terms of significant shifts can be adaptively attained by employing a randomized variant of elimination within our blackbox scheme. Our enhanced rates only depend on the rotting non-stationarity and thus exhibit an interesting phenomenon for this problem where rising rewards do not factor into the difficulty of non-stationarity.