MatrixWorld: A pursuit-evasion platform for safe multi-agent coordination and autocurricula

TL;DR

MatrixWorld is a new multi-agent environment designed to enhance safety in reinforcement learning tasks, supporting pursuit-evasion scenarios with safety constraints, conflict resolution, and serving as a platform for autocurricula research.

Contribution

This work introduces MatrixWorld, a safety-constrained pursuit-evasion platform with novel conflict resolution methods and a co-evolution framework for safe multi-agent learning and autocurricula studies.

Findings

Extended conflict resolution to heterogeneous/adversarial agents

Proposed three conflict resolution strategies

MatrixWorld supports diverse pursuit-evasion variants

Abstract

Multi-agent reinforcement learning (MARL) achieves encouraging performance in solving complex tasks. However, the safety of MARL policies is one critical concern that impedes their real-world applications. Popular multi-agent benchmarks focus on diverse tasks yet provide limited safety support. Therefore, this work proposes a safety-constrained multi-agent environment: MatrixWorld, based on the general pursuit-evasion game. Particularly, a safety-constrained multi-agent action execution model is proposed for the software implementation of safe multi-agent environments based on diverse safety definitions. It (1) extends the vertex conflict among homogeneous / cooperative agents to heterogeneous / adversarial settings, and (2) proposes three types of resolutions for each type of conflict, aiming at providing rational and unbiased feedback for safe MARL. Besides, MatrixWorld is also a lightweight co-evolution framework for the learning of pursuit tasks, evasion tasks, or both, where more pursuit-evasion variants can be designed based on different practical meanings of safety. As a brief survey, we review and analyze the co-evolution mechanism in the multi-agent setting, which clearly reveals its relationships with autocurricula, self-play, arms races, and adversarial learning. Thus, MatrixWorld can also serve as the first environment for autocurricula research, where ideas can be quickly verified and well understood.