Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL

TL;DR

Oryx is a new offline multi-agent reinforcement learning algorithm that improves coordination in complex environments by combining autoregressive policies with implicit constraint Q-learning, achieving state-of-the-art results across diverse benchmarks.

Contribution

The paper introduces Oryx, a novel offline MARL algorithm that effectively handles long-horizon coordination in many-agent settings using a retention-based architecture and implicit constraint Q-learning.

Findings

Achieves state-of-the-art performance on over 80% of tested datasets.

Demonstrates robust generalization across domains with many agents.

Effectively scales to complex, long-horizon tasks.

Abstract

A key challenge in offline multi-agent reinforcement learning (MARL) is achieving effective many-agent multi-step coordination in complex environments. In this work, we propose Oryx, a novel algorithm for offline cooperative MARL to directly address this challenge. Oryx adapts the recently proposed retention-based architecture Sable and combines it with a sequential form of implicit constraint Q-learning (ICQ), to develop a novel offline autoregressive policy update scheme. This allows Oryx to solve complex coordination challenges while maintaining temporal coherence over long trajectories. We evaluate Oryx across a diverse set of benchmarks from prior works -- SMAC, RWARE, and Multi-Agent MuJoCo -- covering tasks of both discrete and continuous control, varying in scale and difficulty. Oryx achieves state-of-the-art performance on more than 80% of the 65 tested datasets, outperforming prior offline MARL methods and demonstrating robust generalisation across domains with many agents and long horizons. Finally, we introduce new datasets to push the limits of many-agent coordination in offline MARL, and demonstrate Oryx's superior ability to scale effectively in such settings.