Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces SUBSAMPLE-MFQ, a scalable multi-agent reinforcement learning algorithm that efficiently learns near-optimal policies by sampling a subset of agents, with convergence guarantees independent of total agent count.

Contribution

The paper proposes a novel subsampling-based mean-field Q-learning algorithm with convergence guarantees, enabling scalable multi-agent RL with polynomial time complexity in the subsample size.

Findings

Algorithm converges to near-optimal policy as subsample size increases.

Convergence rate is independent of total number of agents.

Provides polynomial-time learning method for large multi-agent systems.

Abstract

Designing efficient algorithms for multi-agent reinforcement learning (MARL) is fundamentally challenging because the size of the joint state and action spaces grows exponentially in the number of agents. These difficulties are exacerbated when balancing sequential global decision-making with local agent interactions. In this work, we propose a new algorithm $\texttt{SUBSAMPLE-MFQ}$ ($\textbf{Subsample}$-$\textbf{M}$ean-$\textbf{F}$ield-$\textbf{Q}$-learning) and a decentralized randomized policy for a system with $n$ agents. For any $k\leq n$, our algorithm learns a policy for the system in time polynomial in $k$. We prove that this learned policy converges to the optimal policy on the order of $\tilde{O}(1/\sqrt{k})$ as the number of subsampled agents $k$ increases. In particular, this bound is independent of the number of agents $n$.