From Restless to Contextual: A Thresholding Bandit Reformulation For Finite-horizon Improvement

TL;DR

This paper reformulates restless bandit problems as budgeted thresholding contextual bandits, enabling faster convergence and improved finite-horizon performance with sublinear regret and empirical gains.

Contribution

It introduces a novel reformulation of restless bandits into a simpler thresholding bandit model and proves non-asymptotic optimality for a simplified setting.

Findings

Achieves sublinear regret in a multi-state, heterogeneous setting.

Demonstrates faster convergence than existing algorithms.

Empirically outperforms state-of-the-art methods in large-scale environments.

Abstract

This paper addresses the poor finite-horizon performance of existing online \emph{restless bandit} (RB) algorithms, which stems from the prohibitive sample complexity of learning a full \emph{Markov decision process} (MDP) for each agent. We argue that superior finite-horizon performance requires \emph{rapid convergence} to a \emph{high-quality} policy. Thus motivated, we introduce a reformulation of online RBs as a \emph{budgeted thresholding contextual bandit}, which simplifies the learning problem by encoding long-term state transitions into a scalar reward. We prove the first non-asymptotic optimality of an oracle policy for a simplified finite-horizon setting. We propose a practical learning policy under a heterogeneous-agent, multi-state setting, and show that it achieves a sublinear regret, achieving \emph{faster convergence} than existing methods. This directly translates to higher cumulative reward, as empirically validated by significant gains over state-of-the-art algorithms in large-scale heterogeneous environments. The code is provided in \href{https://github.com/jamie01713/EGT}{github}. Our work provides a new pathway for achieving practical, sample-efficient learning in finite-horizon RBs.