Weakly-Coupled Multi-Action Restless Bandits -- Exponential Convergence in Probability

TL;DR

This paper analyzes a class of multi-action restless bandit problems with weak coupling constraints, proving exponential convergence of policies to optimality as the system size grows, without relying on non-degenerate assumptions.

Contribution

It establishes exponential convergence in probability for general policies in weakly-coupled restless bandits, extending prior results to broader settings without restrictive assumptions.

Findings

Any policy in a broad class converges exponentially to a deterministic process as system size increases.

The convergence rate is exponential and does not depend on non-degenerate assumptions.

A specific policy is proposed that converges exponentially fast to optimality in probability.

Abstract

We study a finite time horizon Markov decision process (MDP) consisting of several groups of multi-action finite-state restless bandit processes, which are identical within each group. The bandit processes into different groups can be rather different. The bandit processes are subject to multiple weakly coupled constraints on their state and action variables. In contrast to prior studies that considered only a few specific policies/algorithms, here, we study the behaviours of the general stochastic process and, most importantly, the design of policies that guarantee its convergence to an ideal trajectory as the problem size increases. We prove that, for any policy in a rather general class, the resulting stochastic process converges in probability to a deterministic process as the system size (measured by the number of bandits) tends to infinity, at an exponential rate. Unlike the previous proofs, our exponential convergence does not rely on any non-degenerate assumptions. It follows that the chosen policy asymptotically approaches optimality (with exponentially diminishing suboptimality) in the size dimension if and only if the deterministic process coincides with optimality. We further propose a policy and prove that, in general, it converges in probability to optimality exponentially fast in the system size.