Sable: a Performant, Efficient and Scalable Sequence Model for MARL

TL;DR

Sable is a new sequence modeling approach for multi-agent reinforcement learning that offers high performance, memory efficiency, and scalability, effectively handling large numbers of agents and long temporal contexts.

Contribution

This paper introduces Sable, a novel, scalable, and memory-efficient sequence model for MARL that adapts retention mechanisms for improved temporal reasoning and performance.

Findings

Outperforms state-of-the-art methods in 34 out of 45 environments

Maintains performance with over a thousand agents

Exhibits linear memory growth with increasing agents

Abstract

As multi-agent reinforcement learning (MARL) progresses towards solving larger and more complex problems, it becomes increasingly important that algorithms exhibit the key properties of (1) strong performance, (2) memory efficiency, and (3) scalability. In this work, we introduce Sable, a performant, memory-efficient, and scalable sequence modeling approach to MARL. Sable works by adapting the retention mechanism in Retentive Networks (Sun et al., 2023) to achieve computationally efficient processing of multi-agent observations with long context memory for temporal reasoning. Through extensive evaluations across six diverse environments, we demonstrate how Sable is able to significantly outperform existing state-of-the-art methods in a large number of diverse tasks (34 out of 45 tested). Furthermore, Sable maintains performance as we scale the number of agents, handling environments with more than a thousand agents while exhibiting a linear increase in memory usage. Finally, we conduct ablation studies to isolate the source of Sable's performance gains and confirm its efficient computational memory usage.