Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO

TL;DR

This paper reports on the Neural MMO challenge at IJCAI 2022, which evaluated the robustness and generalization of multi-agent systems using a complex multi-task environment, highlighting the effectiveness of standard RL methods.

Contribution

It introduces a new benchmark for multi-agent robustness and generalization, along with a comprehensive competition framework and open-source tools for future research.

Findings

Top submissions achieved high success with standard RL methods.

Competitions can effectively benchmark and advance multi-agent algorithms.

Open-source environment and tools will facilitate further research.

Abstract

We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. The competition combines relatively complex environment design with large numbers of agents in the environment. The top submissions demonstrate strong success on this task using mostly standard reinforcement learning (RL) methods combined with domain-specific engineering. We summarize the competition design and results and suggest that, as an academic community, competitions may be a powerful approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.