Transformer-Based Scalable Multi-Agent Reinforcement Learning for Networked Systems with Long-Range Interactions

TL;DR

This paper introduces STACCA, a transformer-based multi-agent reinforcement learning framework that effectively models long-range dependencies and generalizes across different network topologies for large-scale network control tasks.

Contribution

STACCA is a novel transformer-based MARL framework that captures long-range interactions and generalizes across network structures, addressing key limitations of prior methods.

Findings

STACCA outperforms existing methods in epidemic containment tasks.

It demonstrates strong generalization across diverse network topologies.

The framework scales effectively to large networked systems.

Abstract

Multi-agent reinforcement learning (MARL) has shown promise for large-scale network control, yet existing methods face two major limitations. First, they typically rely on assumptions leading to decay properties of local agent interactions, limiting their ability to capture long-range dependencies such as cascading power failures or epidemic outbreaks. Second, most approaches lack generalizability across network topologies, requiring retraining when applied to new graphs. We introduce STACCA (Shared Transformer Actor-Critic with Counterfactual Advantage), a unified transformer-based MARL framework that addresses both challenges. STACCA employs a centralized Graph Transformer Critic to model long-range dependencies and provide system-level feedback, while its shared Graph Transformer Actor learns a generalizable policy capable of adapting across diverse network structures. Further, to improve credit assignment during training, STACCA integrates a novel counterfactual advantage estimator that is compatible with state-value critic estimates. We evaluate STACCA on epidemic containment and rumor-spreading network control tasks, demonstrating improved performance, network generalization, and scalability. These results highlight the potential of transformer-based MARL architectures to achieve scalable and generalizable control in large-scale networked systems.