Actor-Attention-Critic for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces an actor-critic algorithm with an attention mechanism for decentralized multi-agent reinforcement learning, improving scalability and applicability across various cooperative, competitive, and complex environments.

Contribution

It proposes a novel attention-based critic that enhances learning efficiency and flexibility in multi-agent reinforcement learning without assuming specific action spaces or global states.

Findings

Outperforms recent methods in complex environments

Effective in both cooperative and adversarial settings

Scalable to many agents and diverse reward structures

Abstract

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.