Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?

TL;DR

This paper shows that independent learning methods like IPPO can perform as well as or better than joint learning approaches in multi-agent reinforcement learning benchmarks, highlighting robustness and simplicity.

Contribution

The paper demonstrates that independent PPO can match or outperform centralized methods in multi-agent settings, challenging the necessity of joint value functions.

Findings

IPPO performs comparably or better than joint learning methods on SMAC.

IPPO exhibits robustness to environment non-stationarity.

Minimal hyperparameter tuning suffices for strong performance.

Abstract

Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function. In this paper, we demonstrate that, despite its various theoretical shortcomings, Independent PPO (IPPO), a form of independent learning in which each agent simply estimates its local value function, can perform just as well as or better than state-of-the-art joint learning approaches on popular multi-agent benchmark suite SMAC with little hyperparameter tuning. We also compare IPPO to several variants; the results suggest that IPPO's strong performance may be due to its robustness to some forms of environment non-stationarity.