SA-MATD3:Self-attention-based multi-agent continuous control method in cooperative environments

TL;DR

SA-MATD3 introduces a self-attention-based multi-agent reinforcement learning algorithm that enhances learning efficiency and uniformity among agents in cooperative continuous control tasks, outperforming existing methods in multi-agent environments.

Contribution

The paper proposes a novel multi-agent actor-critic structure with self-attention and improved sample utilization, addressing uneven learning and efficiency issues in multi-agent reinforcement learning.

Findings

Outperforms state-of-the-art algorithms in multi-agent tasks.

Achieves higher learning efficiency as the number of agents increases.

Ensures more uniform learning among agents.

Abstract

Cooperative problems under continuous control have always been the focus of multi-agent reinforcement learning. Existing algorithms suffer from the problem of uneven learning degree with the increase of the number of agents. In this paper, a new structure for a multi-agent actor critic is proposed, and the self-attention mechanism is applied in the critic network and the value decomposition method used to solve the uneven problem. The proposed algorithm makes full use of the samples in the replay memory buffer to learn the behavior of a class of agents. First, a new update method is proposed for policy networks that promotes learning efficiency. Second, the utilization of samples is improved, at the same time reflecting the ability of perspective-taking among groups. Finally, the "deceptive signal" in training is eliminated and the learning degree among agents is more uniform than in the existing methods. Multiple experiments were conducted in two typical scenarios of a multi-agent particle environment. Experimental results show that the proposed algorithm can perform better than the state-of-the-art ones, and that it exhibits higher learning efficiency with an increasing number of agents.