SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning

TL;DR

SMACv2 is a new, more challenging benchmark for cooperative multi-agent reinforcement learning that emphasizes the need for complex policies due to increased stochasticity and partial observability.

Contribution

The paper introduces SMACv2, an improved benchmark with procedural scenario generation and partial observability challenges to better evaluate advanced MARL algorithms.

Findings

SMACv2 scenarios require closed-loop policies.

State-of-the-art algorithms perform less well on SMACv2.

SMACv2 addresses limitations of the original SMAC benchmark.

Abstract

The availability of challenging benchmarks has played a key role in the recent progress of machine learning. In cooperative multi-agent reinforcement learning, the StarCraft Multi-Agent Challenge (SMAC) has become a popular testbed for centralised training with decentralised execution. However, after years of sustained improvement on SMAC, algorithms now achieve near-perfect performance. In this work, we conduct new analysis demonstrating that SMAC lacks the stochasticity and partial observability to require complex *closed-loop* policies. In particular, we show that an *open-loop* policy conditioned only on the timestep can achieve non-trivial win rates for many SMAC scenarios. To address this limitation, we introduce SMACv2, a new version of the benchmark where scenarios are procedurally generated and require agents to generalise to previously unseen settings (from the same distribution) during evaluation. We also introduce the extended partial observability challenge (EPO), which augments SMACv2 to ensure meaningful partial observability. We show that these changes ensure the benchmark requires the use of *closed-loop* policies. We evaluate state-of-the-art algorithms on SMACv2 and show that it presents significant challenges not present in the original benchmark. Our analysis illustrates that SMACv2 addresses the discovered deficiencies of SMAC and can help benchmark the next generation of MARL methods. Videos of training are available at https://sites.google.com/view/smacv2.