ARAC: Adaptive Regularized Multi-Agent Soft Actor-Critic in Graph-Structured Adversarial Games

TL;DR

ARAC introduces an adaptive regularization approach combined with graph neural networks to improve coordination and learning efficiency in multi-agent adversarial tasks with sparse rewards.

Contribution

The paper presents a novel ARAC framework that integrates attention-based GNNs with adaptive divergence regularization for enhanced multi-agent reinforcement learning in graph-structured adversarial environments.

Findings

ARAC achieves faster convergence in pursuit and confrontation tasks.

ARAC attains higher success rates compared to baseline methods.

ARAC demonstrates strong scalability with increasing number of agents.

Abstract

In graph-structured multi-agent reinforcement learning (MARL) adversarial tasks such as pursuit and confrontation, agents must coordinate under highly dynamic interactions, where sparse rewards hinder efficient policy learning. We propose Adaptive Regularized Multi-Agent Soft Actor-Critic (ARAC), which integrates an attention-based graph neural network (GNN) for modeling agent dependencies with an adaptive divergence regularization mechanism. The GNN enables expressive representation of spatial relations and state features in graph environments. Divergence regularization can serve as policy guidance to alleviate the sparse reward problem, but it may lead to suboptimal convergence when the reference policy itself is imperfect. The adaptive divergence regularization mechanism enables the framework to exploit reference policies for efficient exploration in the early stages, while gradually reducing reliance on them as training progresses to avoid inheriting their limitations. Experiments in pursuit and confrontation scenarios demonstrate that ARAC achieves faster convergence, higher final success rates, and stronger scalability across varying numbers of agents compared with MARL baselines, highlighting its effectiveness in complex graph-structured environments.