From Theory to Practice with RAVEN-UCB: Addressing Non-Stationarity in Multi-Armed Bandits through Variance Adaptation

TL;DR

RAVEN-UCB is a new algorithm for non-stationary multi-armed bandits that adaptively uses variance information to improve exploration and achieve tighter regret bounds, demonstrating superior performance in dynamic environments.

Contribution

It introduces RAVEN-UCB, a variance-aware, adaptive algorithm with recursive updates, providing both theoretical guarantees and practical improvements over existing methods.

Findings

Achieves tighter regret bounds than UCB1 and UCB-V.

Performs better in non-stationary environments with distributional, periodic, and fluctuating changes.

Demonstrates robustness and efficiency in synthetic and logistics scenarios.

Abstract

The Multi-Armed Bandit (MAB) problem is challenging in non-stationary environments where reward distributions evolve dynamically. We introduce RAVEN-UCB, a novel algorithm that combines theoretical rigor with practical efficiency via variance-aware adaptation. It achieves tighter regret bounds than UCB1 and UCB-V, with gap-dependent regret of order $K \sigma_{\max}^2 \log T / \Delta$ and gap-independent regret of order $\sqrt{K T \log T}$. RAVEN-UCB incorporates three innovations: (1) variance-driven exploration using $\sqrt{\hat{\sigma}_k^2 / (N_k + 1)}$ in confidence bounds, (2) adaptive control via $\alpha_t = \alpha_0 / \log(t + \epsilon)$, and (3) constant-time recursive updates for efficiency. Experiments across non-stationary patterns - distributional changes, periodic shifts, and temporary fluctuations - in synthetic and logistics scenarios demonstrate its superiority over state-of-the-art baselines, confirming theoretical and practical robustness.