PettingZoo: Gym for Multi-Agent Reinforcement Learning

TL;DR

PettingZoo introduces a comprehensive multi-agent RL environment library with a novel API based on the AEC games model, enhancing research accessibility, reproducibility, and conceptual clarity in MARL.

Contribution

The paper presents PettingZoo, a new multi-agent RL library with a unique API and the AEC games model, improving upon existing MARL environment frameworks.

Findings

PettingZoo offers diverse multi-agent environments with a unified API.

The AEC games model addresses conceptual issues in existing MARL game models.

Case studies demonstrate improved clarity and bug detection in MARL research.

Abstract

This paper introduces the PettingZoo library and the accompanying Agent Environment Cycle ("AEC") games model. PettingZoo is a library of diverse sets of multi-agent environments with a universal, elegant Python API. PettingZoo was developed with the goal of accelerating research in Multi-Agent Reinforcement Learning ("MARL"), by making work more interchangeable, accessible and reproducible akin to what OpenAI's Gym library did for single-agent reinforcement learning. PettingZoo's API, while inheriting many features of Gym, is unique amongst MARL APIs in that it's based around the novel AEC games model. We argue, in part through case studies on major problems in popular MARL environments, that the popular game models are poor conceptual models of games commonly used in MARL and accordingly can promote confusing bugs that are hard to detect, and that the AEC games model addresses these problems.