Online Learning of Whittle Indices for Restless Bandits with Non-Stationary Transition Kernels

TL;DR

This paper introduces an adaptive online algorithm for resource allocation in non-stationary restless bandit problems, leveraging Whittle indices with theoretical guarantees and practical efficiency.

Contribution

It proposes a Sliding-Window Online Whittle policy that adapts to unknown, time-varying dynamics with sub-linear regret guarantees and a method to tune window size online.

Findings

Algorithm achieves sub-linear dynamic regret.

Outperforms baselines in non-stationary environments.

Effectively adapts to unknown variation budgets.

Abstract

The restless multi-armed bandit (RMAB) framework is a popular approach to solving resource allocation problems in networked systems. In this paper, we study optimal resource allocation in RMABs facing unknown and non-stationary dynamics. Solving RMABs optimally is known to be PSPACE-hard even with full knowledge of model parameters. While Whittle index policies offer asymptotic optimality with low computational cost, they require access to stationary transition kernels, an unrealistic assumption in many modern networking applications. To address this challenge, we propose a Sliding-Window Online Whittle (SW-Whittle) policy that remains computationally efficient while adapting to time-varying kernels. Through theoretical analysis, we show that our algorithm achieves sub-linear dynamic regret with respect to the number of episodes. We further address the important case where the variation budget is unknown in advance by combining a Bandit-over-Bandit framework with our sliding-window design. In our scheme, window lengths are tuned online as a function of the estimated variation, while Whittle indices are computed via an upper-confidence-bound of the estimated transition kernels and a bilinear optimization routine. Numerical experiments demonstrate that our algorithm consistently outperforms baselines, achieving the lowest cumulative regret across a range of non-stationary environments.