Decentralized MARL for Coarse Correlated Equilibrium in Aggregative Markov Games

TL;DR

This paper introduces a decentralized, model-free V-learning algorithm for aggregative Markov games that efficiently learns approximate coarse correlated equilibria without the curse of multiagents.

Contribution

It proposes an adaptive, stage-based V-learning algorithm that leverages the aggregative structure for decentralized multi-agent reinforcement learning.

Findings

Achieves epsilon-approximate CCE in O(S Amax T5 / epsilon2) episodes.

Avoids the curse of multiagents common in MARL.

Numerical results confirm theoretical guarantees.

Abstract

This paper studies the problem of decentralized learning of Coarse Correlated Equilibrium (CCE) in aggregative Markov games (AMGs), where each agent's instantaneous reward depends only on its own action and an aggregate quantity. Existing CCE learning algorithms for general Markov games are not designed to leverage the aggregative structure, and research on decentralized CCE learning for AMGs remains limited. We propose an adaptive stage-based V-learning algorithm that exploits the aggregative structure under a fully decentralized information setting. Based on the two-timescale idea, the algorithm partitions learning into stages and adjusts stage lengths based on the variability of aggregate signals, while using no-regret updates within each stage. We prove the algorithm achieves an epsilon-approximate CCE in O(S Amax T5 / epsilon2) episodes, avoiding the curse of multiagents which commonly arises in MARL. Numerical results verify the theoretical findings, and the decentralized, model-free design enables easy extension to large-scale multi-agent scenarios.