Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning

TL;DR

This paper introduces OG-MARL, a comprehensive repository of datasets and baselines designed to advance offline multi-agent reinforcement learning research in realistic, complex environments.

Contribution

The authors release OG-MARL, the first standardized benchmark dataset collection with baselines for offline MARL, addressing a key gap in the field.

Findings

Provides diverse datasets mimicking real-world multi-agent systems

Includes baseline algorithms for benchmarking progress

Facilitates research in complex, realistic environments

Abstract

Being able to harness the power of large datasets for developing cooperative multi-agent controllers promises to unlock enormous value for real-world applications. Many important industrial systems are multi-agent in nature and are difficult to model using bespoke simulators. However, in industry, distributed processes can often be recorded during operation, and large quantities of demonstrative data stored. Offline multi-agent reinforcement learning (MARL) provides a promising paradigm for building effective decentralised controllers from such datasets. However, offline MARL is still in its infancy and therefore lacks standardised benchmark datasets and baselines typically found in more mature subfields of reinforcement learning (RL). These deficiencies make it difficult for the community to sensibly measure progress. In this work, we aim to fill this gap by releasing off-the-grid MARL (OG-MARL): a growing repository of high-quality datasets with baselines for cooperative offline MARL research. Our datasets provide settings that are characteristic of real-world systems, including complex environment dynamics, heterogeneous agents, non-stationarity, many agents, partial observability, suboptimality, sparse rewards and demonstrated coordination. For each setting, we provide a range of different dataset types (e.g. Good, Medium, Poor, and Replay) and profile the composition of experiences for each dataset. We hope that OG-MARL will serve the community as a reliable source of datasets and help drive progress, while also providing an accessible entry point for researchers new to the field.