Bounded Memory Adversarial Bandits with Composite Anonymous Delayed Feedback

TL;DR

This paper investigates adversarial bandit problems with composite anonymous delayed feedback, revealing the challenges of non-oblivious settings and proposing algorithms with sublinear regret bounds for bounded memory loss sequences.

Contribution

It introduces a wrapper algorithm achieving sublinear policy regret in non-oblivious delay settings and establishes a matching lower bound, advancing understanding of delayed feedback in adversarial bandits.

Findings

Non-oblivious setting incurs linear regret without assumptions.

Proposed algorithm achieves $o(T)$ policy regret under bounded memory.

Lower bound matches $T^{2/3}$ regret, confirming the difficulty of non-oblivious delays.

Abstract

We study the adversarial bandit problem with composite anonymous delayed feedback. In this setting, losses of an action are split into $d$ components, spreading over consecutive rounds after the action is chosen. And in each round, the algorithm observes the aggregation of losses that come from the latest $d$ rounds. Previous works focus on oblivious adversarial setting, while we investigate the harder non-oblivious setting. We show non-oblivious setting incurs $\Omega(T)$ pseudo regret even when the loss sequence is bounded memory. However, we propose a wrapper algorithm which enjoys $o(T)$ policy regret on many adversarial bandit problems with the assumption that the loss sequence is bounded memory. Especially, for $K$-armed bandit and bandit convex optimization, we have $\mathcal{O}(T^{2/3})$ policy regret bound. We also prove a matching lower bound for $K$-armed bandit. Our lower bound works even when the loss sequence is oblivious but the delay is non-oblivious. It answers the open problem proposed in \cite{wang2021adaptive}, showing that non-oblivious delay is enough to incur $\tilde{\Omega}(T^{2/3})$ regret.