Decentralized Natural Policy Gradient with Variance Reduction for Collaborative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces MDNPG, a decentralized natural policy gradient algorithm with variance reduction, achieving optimal convergence and superior empirical results in multi-agent reinforcement learning with communication constraints.

Contribution

The paper proposes MDNPG, a novel decentralized policy gradient method combining natural gradient, momentum-based variance reduction, and gradient tracking, with proven optimal convergence rates.

Findings

MDNPG achieves $ ilde{O}(n^{-1} ext{poly}(rac{1}{ ext{epsilon}}))$ sample complexity.

MDNPG demonstrates linear speedup over centralized methods.

Empirical results show MDNPG outperforms existing algorithms in multi-agent tasks.

Abstract

This paper studies a policy optimization problem arising from collaborative multi-agent reinforcement learning in a decentralized setting where agents communicate with their neighbors over an undirected graph to maximize the sum of their cumulative rewards. A novel decentralized natural policy gradient method, dubbed Momentum-based Decentralized Natural Policy Gradient (MDNPG), is proposed, which incorporates natural gradient, momentum-based variance reduction, and gradient tracking into the decentralized stochastic gradient ascent framework. The $\mathcal{O}(n^{-1}\epsilon^{-3})$ sample complexity for MDNPG to converge to an $\epsilon$-stationary point has been established under standard assumptions, where $n$ is the number of agents. It indicates that MDNPG can achieve the optimal convergence rate for decentralized policy gradient methods and possesses a linear speedup in contrast to centralized optimization methods. Moreover, superior empirical performance of MDNPG over other state-of-the-art algorithms has been demonstrated by extensive numerical experiments.