Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learning

TL;DR

This paper extends mean-field game theory to asymmetric multi-agent settings, providing methods for symmetrization, approximation bounds, and convergence guarantees, enabling scalable learning in large heterogeneous multi-agent systems.

Contribution

It introduces a framework for extending finite asymmetric games to induced MFGs, with explicit bounds and convergence guarantees for approximate Nash policies.

Findings

Symmetrization of N-player games via Kirszbraun extensions.

Explicit bounds for approximate Nash equilibria in $eta$-symmetric games.

Sample complexity of $ ilde{O}( ext{epsilon}^{-6})$ for learning epsilon-Nash equilibria.

Abstract

Mean-field games (MFG) have become significant tools for solving large-scale multi-agent reinforcement learning problems under symmetry. However, the assumption of exact symmetry limits the applicability of MFGs, as real-world scenarios often feature inherent heterogeneity. Furthermore, most works on MFG assume access to a known MFG model, which might not be readily available for real-world finite-agent games. In this work, we broaden the applicability of MFGs by providing a methodology to extend any finite-player, possibly asymmetric, game to an "induced MFG". First, we prove that $N$-player dynamic games can be symmetrized and smoothly extended to the infinite-player continuum via explicit Kirszbraun extensions. Next, we propose the notion of $\alpha,\beta$-symmetric games, a new class of dynamic population games that incorporate approximate permutation invariance. For $\alpha,\beta$-symmetric games, we establish explicit approximation bounds, demonstrating that a Nash policy of the induced MFG is an approximate Nash of the $N$-player dynamic game. We show that TD learning converges up to a small bias using trajectories of the $N$-player game with finite-sample guarantees, permitting symmetrized learning without building an explicit MFG model. Finally, for certain games satisfying monotonicity, we prove a sample complexity of $\widetilde{\mathcal{O}}(\varepsilon^{-6})$ for the $N$-agent game to learn an $\varepsilon$-Nash up to symmetrization bias. Our theory is supported by evaluations on MARL benchmarks with thousands of agents.