Coordination Matters: Evaluation of Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a coordination-aware evaluation approach for cooperative multi-agent reinforcement learning, revealing how different coordination mechanisms affect performance beyond traditional return metrics.

Contribution

It proposes a new evaluation perspective and a controlled testbed to systematically analyze coordination strategies in MARL.

Findings

Similar returns can mask different coordination mechanisms.

Coordination factors like assignment diversity influence scalability.

Performance depends on assignment pressure and decision opportunities.

Abstract

Cooperative multi-agent reinforcement learning (MARL) benchmarks commonly emphasize aggregate outcomes such as return, success rate, or completion time. While essential, these metrics often fail to reveal how agents coordinate, particularly in settings where agents, tasks, and joint assignment choices scale combinatorially. We propose a coordination-aware evaluation perspective that supplements return with process-level diagnostics. We instantiate this perspective using STAT, a controlled commitment-constrained spatial task-allocation testbed that systematically varies agents, tasks, and environment size while holding observation access and task rules fixed. We evaluate six representative value-based MARL methods across varying levels of centralization. Our results show that similar return trends can reflect distinct coordination mechanisms, including differences in redundant assignment, assignment diversity, and task-completion efficiency. We find that in commitment-constrained task allocation, performance under scale is shaped not only by nominal action-space size, but also by assignment pressure, sparse decision opportunities, and redundant choices among interdependent agents. Our findings motivate coordination-aware evaluation as a necessary complement to return-based benchmarking for cooperative MARL.