Model Predictive Control is Almost Optimal for Restless Bandit

TL;DR

This paper introduces a model predictive control approach for restless bandit problems, providing near-optimal solutions with quantifiable sub-optimality bounds that perform well in practice and are easy to implement.

Contribution

The paper proposes a novel MPC-based policy for RMABs with minimal assumptions, offering explicit sub-optimality bounds and a framework adaptable to broader constrained MDPs.

Findings

Sub-optimality gap is O(1/√N) in general

Exponential decay of sub-optimality under local-stability

Policy performs well compared to state-of-the-art methods

Abstract

We consider the discrete time infinite horizon average reward restless markovian bandit (RMAB) problem. We propose a \emph{model predictive control} based non-stationary policy with a rolling computational horizon $\tau$. At each time-slot, this policy solves a $\tau$ horizon linear program whose first control value is kept as a control for the RMAB. Our solution requires minimal assumptions and quantifies the loss in optimality in terms of $\tau$ and the number of arms, $N$. We show that its sub-optimality gap is $O(1/\sqrt{N})$ in general, and $\exp(-\Omega(N))$ under a local-stability condition. Our proof is based on a framework from dynamic control known as \emph{dissipativity}. Our solution easy to implement and performs very well in practice when compared to the state of the art. Further, both our solution and our proof methodology can easily be generalized to more general constrained MDP settings and should thus, be of great interest to the burgeoning RMAB community.