MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learning

TL;DR

MA2CL introduces a contrastive learning framework that enhances multi-agent reinforcement learning by reconstructing masked agent observations, leading to improved cooperation, performance, and sample efficiency in vision-based scenarios.

Contribution

The paper proposes MA2CL, a novel masked attentive contrastive learning method that incorporates agent-level information for better representation learning in MARL.

Findings

Significantly improves MARL performance in various scenarios.

Enhances sample efficiency of reinforcement learning algorithms.

Outperforms existing methods in vision-based and state-based tasks.

Abstract

Recent approaches have utilized self-supervised auxiliary tasks as representation learning to improve the performance and sample efficiency of vision-based reinforcement learning algorithms in single-agent settings. However, in multi-agent reinforcement learning (MARL), these techniques face challenges because each agent only receives partial observation from an environment influenced by others, resulting in correlated observations in the agent dimension. So it is necessary to consider agent-level information in representation learning for MARL. In this paper, we propose an effective framework called \textbf{M}ulti-\textbf{A}gent \textbf{M}asked \textbf{A}ttentive \textbf{C}ontrastive \textbf{L}earning (MA2CL), which encourages learning representation to be both temporal and agent-level predictive by reconstructing the masked agent observation in latent space. Specifically, we use an attention reconstruction model for recovering and the model is trained via contrastive learning. MA2CL allows better utilization of contextual information at the agent level, facilitating the training of MARL agents for cooperation tasks. Extensive experiments demonstrate that our method significantly improves the performance and sample efficiency of different MARL algorithms and outperforms other methods in various vision-based and state-based scenarios. Our code can be found in \url{https://github.com/ustchlsong/MA2CL}