GTDE: Grouped Training with Decentralized Execution for Multi-agent Actor-Critic

TL;DR

This paper introduces GTDE, a novel multi-agent training paradigm that groups agents based on observation history, eliminating the need for centralized modules and significantly improving performance in large-scale multi-agent systems.

Contribution

GTDE is a new training framework that uses adaptive grouping and end-to-end training to enhance scalability and performance without centralized coordination.

Findings

Increased total reward by 382% in a 495-agent cooperative environment.

Achieved 100% win rate in a 64-agent competitive environment.

Effectively handles large-scale multi-agent systems with local information only.

Abstract

The rapid advancement of multi-agent reinforcement learning (MARL) has given rise to diverse training paradigms to learn the policies of each agent in the multi-agent system. The paradigms of decentralized training and execution (DTDE) and centralized training with decentralized execution (CTDE) have been proposed and widely applied. However, as the number of agents increases, the inherent limitations of these frameworks significantly degrade the performance metrics, such as win rate, total reward, etc. To reduce the influence of the increasing number of agents on the performance metrics, we propose a novel training paradigm of grouped training decentralized execution (GTDE). This framework eliminates the need for a centralized module and relies solely on local information, effectively meeting the training requirements of large-scale multi-agent systems. Specifically, we first introduce an adaptive grouping module, which divides each agent into different groups based on their observation history. To implement end-to-end training, GTDE uses Gumbel-Sigmoid for efficient point-to-point sampling on the grouping distribution while ensuring gradient backpropagation. To adapt to the uncertainty in the number of members in a group, two methods are used to implement a group information aggregation module that merges member information within the group. Empirical results show that in a cooperative environment with 495 agents, GTDE increased the total reward by an average of 382\% compared to the baseline. In a competitive environment with 64 agents, GTDE achieved a 100\% win rate against the baseline.