On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learning

TL;DR

This paper introduces new decentralized, sample-efficient model-free algorithms for multi-agent reinforcement learning, addressing the exponential complexity challenge and enabling agents to learn equilibria without communication.

Contribution

It proposes stage-based V-learning for correlated equilibria and a decentralized policy gradient with variance reduction for Nash equilibria, simplifying design and analysis.

Findings

Algorithms are decentralized, requiring only local information.

Significant reduction in sample complexity compared to prior methods.

Numerical simulations support theoretical results.

Abstract

Multi-agent reinforcement learning (MARL) algorithms often suffer from an exponential sample complexity dependence on the number of agents, a phenomenon known as \emph{the curse of multiagents}. In this paper, we address this challenge by investigating sample-efficient model-free algorithms in \emph{decentralized} MARL, and aim to improve existing algorithms along this line. For learning (coarse) correlated equilibria in general-sum Markov games, we propose \emph{stage-based} V-learning algorithms that significantly simplify the algorithmic design and analysis of recent works, and circumvent a rather complicated no-\emph{weighted}-regret bandit subroutine. For learning Nash equilibria in Markov potential games, we propose an independent policy gradient algorithm with a decentralized momentum-based variance reduction technique. All our algorithms are decentralized in that each agent can make decisions based on only its local information. Neither communication nor centralized coordination is required during learning, leading to a natural generalization to a large number of agents. We also provide numerical simulations to corroborate our theoretical findings.