Enhancing Multi-Agent Collaboration with Attention-Based Actor-Critic Policies

TL;DR

This paper presents TAAC, a novel reinforcement learning algorithm with attention mechanisms for improved multi-agent collaboration, demonstrating superior performance in simulated soccer tasks compared to existing methods.

Contribution

Introduces TAAC, integrating multi-headed attention in actor-critic frameworks for dynamic inter-agent communication and role diversity in cooperative multi-agent systems.

Findings

TAAC outperforms benchmark algorithms in simulated soccer.

Enhanced collaborative behaviors observed with TAAC.

TAAC achieves higher win rates and better tactical coordination.

Abstract

This paper introduces Team-Attention-Actor-Critic (TAAC), a reinforcement learning algorithm designed to enhance multi-agent collaboration in cooperative environments. TAAC employs a Centralized Training/Centralized Execution scheme incorporating multi-headed attention mechanisms in both the actor and critic. This design facilitates dynamic, inter-agent communication, allowing agents to explicitly query teammates, thereby efficiently managing the exponential growth of joint-action spaces while ensuring a high degree of collaboration. We further introduce a penalized loss function which promotes diverse yet complementary roles among agents. We evaluate TAAC in a simulated soccer environment against benchmark algorithms representing other multi-agent paradigms, including Proximal Policy Optimization and Multi-Agent Actor-Attention-Critic. We find that TAAC exhibits superior performance and enhanced collaborative behaviors across a variety of metrics (win rates, goal differentials, Elo ratings, inter-agent connectivity, balanced spatial distributions, and frequent tactical interactions such as ball possession swaps).