Finite-Time Regret Analysis of Retry-Aware Bandits

TL;DR

This paper analyzes a retry-aware bandit algorithm called ReMax, providing the first sublinear regret bounds and exploring its exploration-exploitation behavior compared to Thompson sampling.

Contribution

It characterizes the optimal ReMax distribution for Gaussian rewards, proves sublinear regret bounds, and explains its unique exploration properties.

Findings

ReMax often outperforms KL-UCB and Thompson sampling under mild underestimation.

Posterior-variance scaling empirically mitigates severe underestimation.

ReMax can be more exploitative than Thompson sampling.

Abstract

We study a stochastic bandit algorithm motivated by retry-aware objectives that value the best outcome among multiple attempts, such as pass@$k$ and max@$k$. Given a posterior over arm values, ReMax chooses a sampling distribution that maximizes the posterior expected maximum reward over $M$ virtual draws. Although this objective was introduced in reinforcement learning as an exploration mechanism under uncertainty, its regret properties in bandit problems have remained unclear. For Gaussian rewards and the first nontrivial case $M=2$, we characterize the optimal ReMax distribution through an expected-improvement balance condition and prove the first sublinear regret bound for ReMax. Our analysis separates the usual saturation behavior of suboptimal arms from a ReMax-specific underestimation effect, in which the optimal arm may be sampled too rarely after an unfavorable estimate. This explains why ReMax can be more exploitative than Thompson sampling (TS) and why its regret analysis is technically delicate. Experiments support this picture: ReMax often outperforms KL-UCB and Thompson sampling under mild underestimation, while posterior-variance scaling empirically mitigates severe underestimation.