Projection-based Lyapunov method for fully heterogeneous weakly-coupled MDPs

TL;DR

This paper introduces a projection-based Lyapunov method to address the challenge of heterogeneity in weakly-coupled Markov decision processes, achieving asymptotic optimality as the number of arms grows large.

Contribution

It presents the first asymptotic optimality result for fully heterogeneous WCMDPs using a novel Lyapunov function construction.

Findings

Achieves an $O(1/ oot N)$ optimality gap for large $N$

First asymptotic optimality result for fully heterogeneous WCMDPs

Introduces a projection-based Lyapunov approach for convergence certification

Abstract

Heterogeneity poses a fundamental challenge for many real-world large-scale decision-making problems but remains largely understudied. In this paper, we study the fully heterogeneous setting of a prominent class of such problems, known as weakly-coupled Markov decision processes (WCMDPs). Each WCMDP consists of $N$ arms (or subproblems), which have distinct model parameters in the fully heterogeneous setting, leading to the curse of dimensionality when $N$ is large. We show that, under mild assumptions, an efficiently computable policy achieves an $O(1/\sqrt{N})$ optimality gap in the long-run average reward per arm for fully heterogeneous WCMDPs as $N$ becomes large. This is the first asymptotic optimality result for fully heterogeneous average-reward WCMDPs. Our main technical innovation is the construction of projection-based Lyapunov functions that certify the convergence of rewards and costs to an optimal region, even under full heterogeneity.