Minimizing Cost Rather Than Maximizing Reward in Restless Multi-Armed Bandits

TL;DR

This paper introduces a new minimization framework for Restless Multi-Armed Bandits focusing on reward thresholds and cost minimization, extending the Whittle index approach and analyzing heuristic solutions.

Contribution

It formulates a constrained minimization problem for RMABs, develops a Whittle index for this setting, and compares heuristics for solving the minimization problem.

Findings

Whittle index for minimization can be derived from maximization index.

Minimization problem is PSPACE-hard even approximately.

Heuristics perform variably, with some achieving optimal solutions and others failing.

Abstract

Restless Multi-Armed Bandits (RMABs) offer a powerful framework for solving resource constrained maximization problems. However, the formulation can be inappropriate for settings where the limiting constraint is a reward threshold rather than a budget. We introduce a constrained minimization problem for RMABs that balances the goal of achieving a reward threshold while minimizing total cost. We show that even a bi-criteria approximate version of the problem is PSPACE-hard. Motivated by the hardness result, we define a decoupled problem, indexability and a Whittle index for the minimization problem, mirroring the corresponding concepts for the maximization problem. Further, we show that the Whittle index for the minimization problem can easily be computed from the Whittle index for the maximization problem. Consequently, Whittle index results on RMAB instances for the maximization problem give Whittle index results for the minimization problem. Despite the similarities between the minimization and maximization problems, solving the minimization problem is not as simple as taking direct analogs of the heuristics for the maximization problem. We give an example of an RMAB for which the greedy Whittle index heuristic achieves the optimal solution for the maximization problem, while the analogous heuristic yields the worst possible solution for the minimization problem. In light of this, we present and compare several heuristics for solving the minimization problem on real and synthetic data. Our work suggests the importance of continued investigation into the minimization problem.