Two families of indexable partially observable restless bandits and Whittle index computation

TL;DR

This paper studies two types of partially observable restless bandits, establishes indexability, derives a closed-form Whittle index for one, and proposes an efficient algorithm for the other, demonstrating superior performance in numerical experiments.

Contribution

It introduces two new models of partially observable restless bandits, proves their indexability, and provides methods for computing the Whittle index, including a closed-form expression and an efficient algorithm.

Findings

Whittle index policy outperforms myopic policy.

Proposed algorithms are computationally efficient.

Numerical results show near-optimal performance.

Abstract

We consider the restless bandits with general state space under partial observability with two observational models: first, the state of each bandit is not observable at all, and second, the state of each bandit is observable only if it is chosen. We assume both models satisfy the restart property under which we prove indexability of the models and propose the Whittle index policy as the solution. For the first model, we derive a closed-form expression for the Whittle index. For the second model, we propose an efficient algorithm to compute the Whittle index by exploiting the qualitative properties of the optimal policy. We present detailed numerical experiments for multiple instances of machine maintenance problem. The result indicates that the Whittle index policy outperforms myopic policy and can be close to optimal in different setups.