Improved Best-of-Both-Worlds Regret for Bandits with Delayed Feedback

TL;DR

This paper introduces a new algorithm for multi-armed bandits with delayed feedback that nearly matches the best possible regret bounds in both stochastic and adversarial environments, improving upon prior methods.

Contribution

A novel algorithm that achieves near-optimal regret bounds in both stochastic and adversarial bandit settings with delays, matching known lower bounds up to logarithmic factors.

Findings

Achieves adversarial regret of (\u221a{KT} + D)

Provides stochastic regret bounds matching lower bounds under delays

First BoBW algorithm to match lower bounds in both regimes with delays

Abstract

We study the multi-armed bandit problem with adversarially chosen delays in the Best-of-Both-Worlds (BoBW) framework, which aims to achieve near-optimal performance in both stochastic and adversarial environments. While prior work has made progress toward this goal, existing algorithms suffer from significant gaps to the known lower bounds, especially in the stochastic settings. Our main contribution is a new algorithm that, up to logarithmic factors, matches the known lower bounds in each setting individually. In the adversarial case, our algorithm achieves regret of $\widetilde{O}(\sqrt{KT} + \sqrt{D})$, which is optimal up to logarithmic terms, where $T$ is the number of rounds, $K$ is the number of arms, and $D$ is the cumulative delay. In the stochastic case, we provide a regret bound which scale as $\sum_{i:\Delta_i>0}\left(\log T/\Delta_i\right) + \frac{1}{K}\sum \Delta_i \sigma_{max}$, where $\Delta_i$ is the sub-optimality gap of arm $i$ and $\sigma_{\max}$ is the maximum number of missing observations. To the best of our knowledge, this is the first BoBW algorithm to simultaneously match the lower bounds in both stochastic and adversarial regimes in delayed environment. Moreover, even beyond the BoBW setting, our stochastic regret bound is the first to match the known lower bound under adversarial delays, improving the second term over the best known result by a factor of $K$.