Lagrangian Index Policy for Restless Bandits with Average Reward

TL;DR

This paper introduces the Lagrangian Index Policy (LIP) for restless bandits, compares it with the Whittle Index Policy, and develops reinforcement learning algorithms for online implementation, demonstrating its efficiency and theoretical properties.

Contribution

It proposes the LIP as a robust alternative to WIP, provides analytical calculations for specific models, and introduces memory-efficient RL algorithms for LIP in a model-free setting.

Findings

LIP performs well even when WIP performs poorly.

Reinforcement learning algorithms for LIP require less memory.

Analytical calculation of Lagrangian index for restart models.

Abstract

We study the Lagrangian Index Policy (LIP) for restless multi-armed bandits with long-run average reward. In particular, we compare the performance of LIP with the performance of the Whittle Index Policy (WIP), both heuristic policies known to be asymptotically optimal under certain natural conditions. Even though in most cases their performances are very similar, in the cases when WIP shows bad performance, LIP continues to perform very well. We then propose reinforcement learning algorithms, both tabular and NN-based, to obtain online learning schemes for LIP in the model-free setting. The proposed reinforcement learning schemes for LIP require significantly less memory than the analogous schemes for WIP. We calculate analytically the Lagrangian index for the restart model, which applies to the optimal web crawling and the minimization of the weighted age of information. We also give a new proof of asymptotic optimality in case of homogeneous arms as the number of arms goes to infinity, based on exchangeability and de Finetti's theorem.