Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learning

TL;DR

This paper introduces GMFS, a scalable framework for cooperative multi-agent reinforcement learning with heterogeneous interactions, using graphon mean-field subsampling to reduce complexity and improve performance.

Contribution

The paper proposes GMFS, a novel subsampling approach that approximates graphon-weighted mean-field interactions, enabling scalable and near-optimal cooperative MARL with heterogeneous agents.

Findings

GMFS achieves near-optimal performance in robotic coordination tasks.

The sample complexity of GMFS scales polynomially with the subsampling parameter.

Theoretical analysis shows an optimality gap of O(1/√κ).

Abstract

Coordinating large populations of interacting agents is a central challenge in multi-agent reinforcement learning (MARL), where the size of the joint state-action space scales exponentially with the number of agents. Mean-field methods alleviate this burden by aggregating agent interactions, but these approaches assume homogeneous interactions. Recent graphon-based frameworks capture heterogeneity, but are computationally expensive as the number of agents grows. Therefore, we introduce $\texttt{GMFS}$, a $\textbf{G}$raphon $\textbf{M}$ean-$\textbf{F}$ield $\textbf{S}$ubsampling framework for scalable cooperative MARL with heterogeneous agent interactions. By subsampling $\kappa$ agents according to interaction strength, we approximate the graphon-weighted mean-field and learn a policy with sample complexity $\mathrm{poly}(\kappa)$ and optimality gap $O(1/\sqrt{\kappa})$. We verify our theory with numerical simulations in robotic coordination, showing that $\texttt{GMFS}$ achieves near-optimal performance.