Tabular and Deep Learning for the Whittle Index

TL;DR

This paper introduces two reinforcement learning algorithms, QWI and QWINN, to learn the Whittle index for RMABPs, demonstrating convergence and scalability advantages over existing methods.

Contribution

The paper presents the first convergence proof for a tabular and a deep RL algorithm for the Whittle index, with scalable neural network implementation for large state spaces.

Findings

QWI converges to true Whittle indices.

QWINN scales to large state spaces effectively.

Both algorithms outperform existing methods in convergence speed.

Abstract

The Whittle index policy is a heuristic that has shown remarkably good performance (with guaranteed asymptotic optimality) when applied to the class of problems known as Restless Multi-Armed Bandit Problems (RMABPs). In this paper we present QWI and QWINN, two reinforcement learning algorithms, respectively tabular and deep, to learn the Whittle index for the total discounted criterion. The key feature is the use of two time-scales, a faster one to update the state-action Q -values, and a relatively slower one to update the Whittle indices. In our main theoretical result we show that QWI, which is a tabular implementation, converges to the real Whittle indices. We then present QWINN, an adaptation of QWI algorithm using neural networks to compute the Q -values on the faster time-scale, which is able to extrapolate information from one state to another and scales naturally to large state-space environments. For QWINN, we show that all local minima of the Bellman error are locally stable equilibria, which is the first result of its kind for DQN-based schemes. Numerical computations show that QWI and QWINN converge faster than the standard Q -learning algorithm, neural-network based approximate Q-learning and other state of the art algorithms.