Towards Efficient Collaboration via Graph Modeling in Reinforcement Learning

TL;DR

This paper introduces a novel graph-based transformer architecture, $f$-MAT, to improve multi-agent reinforcement learning by enabling efficient communication and coordination among neighboring agents during both training and execution.

Contribution

The paper proposes the $f$-MAT model, a factor-based transformer architecture that enhances multi-agent collaboration through efficient message passing and parallel decision-making.

Findings

$f$-MAT outperforms baseline methods in traffic scheduling.

It demonstrates superior performance in power control tasks.

The approach effectively handles complex collaborative problems.

Abstract

In multi-agent reinforcement learning, a commonly considered paradigm is centralized training with decentralized execution. However, in this framework, decentralized execution restricts the development of coordinated policies due to the local observation limitation. In this paper, we consider the cooperation among neighboring agents during execution and formulate their interactions as a graph. Thus, we introduce a novel encoder-decoder architecture named Factor-based Multi-Agent Transformer ($f$-MAT) that utilizes a transformer to enable communication between neighboring agents during both training and execution. By dividing agents into different overlapping groups and representing each group with a factor, $f$-MAT achieves efficient message passing and parallel action generation through factor-based attention layers. Empirical results in networked systems such as traffic scheduling and power control demonstrate that $f$-MAT achieves superior performance compared to strong baselines, thereby paving the way for handling complex collaborative problems.