Trust Region Bounds for Decentralized PPO Under Non-stationarity

TL;DR

This paper develops trust region bounds for decentralized policy optimization in multi-agent reinforcement learning, ensuring performance guarantees even under non-stationary dynamics, and explains the success of recent actor-critic methods like IPPO and MAPPO.

Contribution

It introduces a theoretical framework for trust region bounds in decentralized MARL, especially under non-stationarity, and provides a principled way to enforce these bounds via ratio clipping based on agent count.

Findings

Trust region bounds hold under non-stationarity in decentralized MARL.

Enforcing trust regions via ratio clipping explains the success of IPPO and MAPPO.

Empirical results validate the theoretical analysis and hyperparameter tuning based on agent number.

Abstract

We present trust region bounds for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary. This new analysis provides a theoretical understanding of the strong performance of two recent actor-critic methods for MARL, which both rely on independent ratios, i.e., computing probability ratios separately for each agent's policy. We show that, despite the non-stationarity that independent ratios cause, a monotonic improvement guarantee still arises as a result of enforcing the trust region constraint over all decentralized policies. We also show this trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training, providing a theoretical foundation for proximal ratio clipping. Finally, our empirical results support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training, and tuning the hyperparameters with regards to the number of agents, as predicted by our theoretical analysis.