Sample-Efficient Multi-Agent RL: An Optimization Perspective

TL;DR

This paper introduces a new complexity measure called MADC for multi-agent reinforcement learning in general-sum Markov Games, and proposes a unified, sample-efficient algorithmic framework that works across different equilibrium concepts and is practical to implement.

Contribution

It defines MADC as a novel complexity measure and develops the first unified, sample-efficient algorithmic framework for MARL that handles multiple equilibrium types with practical implementation advantages.

Findings

Achieves sample efficiency in learning Nash, Coarse Correlated, and Correlated Equilibria.

Provides sublinear regret comparable to existing methods.

Simplifies the optimization process for equilibrium computation.

Abstract

We study multi-agent reinforcement learning (MARL) for the general-sum Markov Games (MGs) under the general function approximation. In order to find the minimum assumption for sample-efficient learning, we introduce a novel complexity measure called the Multi-Agent Decoupling Coefficient (MADC) for general-sum MGs. Using this measure, we propose the first unified algorithmic framework that ensures sample efficiency in learning Nash Equilibrium, Coarse Correlated Equilibrium, and Correlated Equilibrium for both model-based and model-free MARL problems with low MADC. We also show that our algorithm provides comparable sublinear regret to the existing works. Moreover, our algorithm combines an equilibrium-solving oracle with a single objective optimization subprocedure that solves for the regularized payoff of each deterministic joint policy, which avoids solving constrained optimization problems within data-dependent constraints (Jin et al. 2020; Wang et al. 2023) or executing sampling procedures with complex multi-objective optimization problems (Foster et al. 2023), thus being more amenable to empirical implementation.