STAIRS-Former: Spatio-Temporal Attention with Interleaved Recursive Structure Transformer for Offline Multi-task Multi-agent Reinforcement Learning

TL;DR

STAIRS-Former introduces a hierarchical transformer architecture with interleaved spatial and temporal attention, improving coordination and long-term dependency modeling in offline multi-agent multi-task reinforcement learning, leading to state-of-the-art results.

Contribution

It proposes a novel hierarchical transformer architecture with token dropout for better coordination and generalization in offline multi-agent multi-task RL.

Findings

Outperforms prior methods on SMAC, SMAC-v2, MPE, and MaMuJoCo benchmarks.

Achieves new state-of-the-art performance across multiple multi-agent tasks.

Demonstrates robustness and improved generalization with token dropout.

Abstract

Offline multi-agent reinforcement learning (MARL) with multi-task datasets is challenging due to varying numbers of agents across tasks and the need to generalize to unseen scenarios. Prior works employ transformers with observation tokenization and hierarchical skill learning to address these issues. However, they underutilize the transformer attention mechanism for inter-agent coordination and rely on a single history token, which limits their ability to capture long-horizon temporal dependencies in partially observable MARL settings. In this paper, we propose STAIRS-Former, a transformer architecture augmented with spatial and temporal hierarchies that enables effective attention over critical tokens while capturing long interaction histories. We further introduce token dropout to enhance robustness and generalization across varying agent populations. Extensive experiments on diverse multi-agent benchmarks, including SMAC, SMAC-v2, MPE, and MaMuJoCo, with multi-task datasets demonstrate that STAIRS-Former consistently outperforms prior methods and achieves new state-of-the-art performance.