MACTAS: Self-Attention-Based Inter-Agent Communication in Multi-Agent Reinforcement Learning with Action-Value Function Decomposition

TL;DR

This paper introduces a differentiable self-attention-based communication method for multi-agent reinforcement learning that is scalable, integrates with existing algorithms, and achieves state-of-the-art results on benchmark tasks.

Contribution

It presents a novel, fully differentiable self-attention communication mechanism that is scalable and compatible with action-value function decomposition in MARL.

Findings

Achieves state-of-the-art performance on SMACv2 benchmark

Scalable to large multi-agent systems due to fixed parameter count

Seamlessly integrates with existing action-value decomposition algorithms

Abstract

Communication is essential for the collective execution of complex tasks by human agents, motivating interest in communication mechanisms for multi-agent reinforcement learning (MARL). However, existing communication protocols in MARL are often complex and non-differentiable. In this work, we introduce a self-attention-based communication method that exchanges information between the agents in MARL. Our proposed approach is fully differentiable, allowing agents to learn to generate messages in a reward-driven manner. The method can be seamlessly integrated with any action-value function decomposition algorithm and can be viewed as an orthogonal extension of such decompositions. Notably, it includes a fixed number of trainable parameters, independent of the number of agents, which makes it scalable to large systems. Experimental results on the SMACv2 benchmark demonstrate the effectiveness of our approach, which achieves state-of-the-art performance on a number of maps. makes it scalable to large systems. Experimental results on the SMACv2 benchmark demonstrate the effectiveness of our approach, which achieves state-of-the-art performance on a number of maps.