Model Predictive Control is almost Optimal for Heterogeneous Restless Multi-armed Bandits

TL;DR

This paper demonstrates that a model predictive control approach with LP updates nearly optimally solves the infinite horizon heterogeneous restless multi-armed bandit problem, providing strong theoretical guarantees and practical efficiency.

Contribution

It extends the LP-update policy to heterogeneous RMABs, proving an $O(rac{ ext{log} N}{ oot{N}})$ optimality gap and connecting it to LP-index policies.

Findings

LP-update policy achieves near-optimality in heterogeneous RMABs.

The policy is computationally efficient with small planning horizon $ au=5$.

Theoretical guarantees generalize to weakly coupled Markov Decision Processes.

Abstract

We consider a general infinite horizon Heterogeneous Restless multi-armed Bandit (RMAB). Heterogeneity is a fundamental problem for many real-world systems largely because it resists many concentration arguments. In this paper, we assume that each of the $N$ arms can have different model parameters. We show that, under a mild assumption of uniform ergodicity, a natural finite-horizon LP-update policy with randomized rounding, that was originally proposed for the homogeneous case, achieves an $O(\log N\sqrt{1/N})$ optimality gap in infinite time average reward problems for fully heterogeneous RMABs. In doing so, we show results that provide strong theoretical guarantees on a well-known algorithm that works very well in practice. The LP-update policy is a model predictive approach that computes a decision at time $t$ by planing over a time-horizon $\{t\dots t+\tau\}$. Our simulation section demonstrates that our algorithm works extremely well even when $\tau$ is very small and set to $5$, which makes it computationally efficient. Our theoretical results draw on techniques from the model predictive control literature by invoking the concept of \emph{dissipativity} and generalize quite easily to the more general weakly coupled heterogeneous Markov Decision Process setting. In addition, we draw a parallel between our own policy and the LP-index policy by showing that the LP-index policy corresponds to $\tau=1$. We describe where the latter's shortcomings arise from and how under our mild assumption we are able to address these shortcomings. The proof of our main theorem answers an open problem posed by (Brown et al 2020), paving the way for several new questions on the LP-update policies.