Optimistic Whittle Index Policy: Online Learning for Restless Bandits

TL;DR

This paper introduces UCWhittle, an online learning algorithm for restless bandits that uses an upper confidence bound approach to estimate transition dynamics and compute optimistic Whittle indices, achieving sublinear regret.

Contribution

It presents the first online learning algorithm for RMABs based on Whittle index policy with UCB, addressing unknown transition dynamics.

Findings

UCWhittle achieves sublinear regret of O(H√T log T).

It outperforms existing online learning baselines in three domains.

Demonstrated effectiveness on a real-world maternal and childcare dataset.

Abstract

Restless multi-armed bandits (RMABs) extend multi-armed bandits to allow for stateful arms, where the state of each arm evolves restlessly with different transitions depending on whether that arm is pulled. Solving RMABs requires information on transition dynamics, which are often unknown upfront. To plan in RMAB settings with unknown transitions, we propose the first online learning algorithm based on the Whittle index policy, using an upper confidence bound (UCB) approach to learn transition dynamics. Specifically, we estimate confidence bounds of the transition probabilities and formulate a bilinear program to compute optimistic Whittle indices using these estimates. Our algorithm, UCWhittle, achieves sublinear $O(H \sqrt{T \log T})$ frequentist regret to solve RMABs with unknown transitions in $T$ episodes with a constant horizon $H$. Empirically, we demonstrate that UCWhittle leverages the structure of RMABs and the Whittle index policy solution to achieve better performance than existing online learning baselines across three domains, including one constructed from a real-world maternal and childcare dataset.