ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination

TL;DR

ZSC-Eval is a comprehensive toolkit and benchmark designed to evaluate zero-shot coordination algorithms in multi-agent reinforcement learning, addressing distribution gaps and providing new metrics for better assessment.

Contribution

The paper introduces ZSC-Eval, the first dedicated evaluation toolkit and benchmark for ZSC algorithms, including new partner generation, selection, and performance measurement methods.

Findings

Benchmarking in Overcooked and Google Research Football environments.

Empirical analysis reveals strengths and weaknesses of current ZSC algorithms.

Human experiments validate the evaluation metrics' alignment with human judgment.

Abstract

Zero-shot coordination (ZSC) is a new cooperative multi-agent reinforcement learning (MARL) challenge that aims to train an ego agent to work with diverse, unseen partners during deployment. The significant difference between the deployment-time partners' distribution and the training partners' distribution determined by the training algorithm makes ZSC a unique out-of-distribution (OOD) generalization challenge. The potential distribution gap between evaluation and deployment-time partners leads to inadequate evaluation, which is exacerbated by the lack of appropriate evaluation metrics. In this paper, we present ZSC-Eval, the first evaluation toolkit and benchmark for ZSC algorithms. ZSC-Eval consists of: 1) Generation of evaluation partner candidates through behavior-preferring rewards to approximate deployment-time partners' distribution; 2) Selection of evaluation partners by Best-Response Diversity (BR-Div); 3) Measurement of generalization performance with various evaluation partners via the Best-Response Proximity (BR-Prox) metric. We use ZSC-Eval to benchmark ZSC algorithms in Overcooked and Google Research Football environments and get novel empirical findings. We also conduct a human experiment of current ZSC algorithms to verify the ZSC-Eval's consistency with human evaluation. ZSC-Eval is now available at https://github.com/sjtu-marl/ZSC-Eval.