GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization

TL;DR

GRASP introduces a novel multi-agent optimization framework that actively aligns gradients to stabilize policy evolution, improving convergence speed and collaboration in complex environments.

Contribution

It proposes a new active perception mechanism using consensus gradients and proves equilibrium existence with Kakutani Fixed-Point Theorem.

Findings

Demonstrates scalability on SMAC and GRF benchmarks.

Achieves promising performance and faster convergence.

Provides theoretical guarantees for equilibrium stability.

Abstract

Non-stationarity arises from concurrent policy updates and leads to persistent environmental fluctuations. Existing approaches like Centralized Training with Decentralized Execution (CTDE) and sequential update schemes mitigate this issue. However, since the perception of the policies of other agents remains dependent on sampling environmental interaction data, the agent essentially operates in a passive perception state. This inevitably triggers equilibrium oscillations and significantly slows the convergence speed of the system. To address this issue, we propose Gradient Realignment via Active Shared Perception (GRASP), a novel framework that defines generalized Bellman equilibrium as a stable objective for policy evolution. The core mechanism of GRASP involves utilizing the independent gradients of agents to derive a defined consensus gradient, enabling agents to actively perceive policy updates and optimize team collaboration. Theoretically, we leverage the Kakutani Fixed-Point Theorem to prove that the consensus direction $u^*$ guarantees the existence and attainability of this equilibrium. Extensive experiments on StarCraft II Multi-Agent Challenge (SMAC) and Google Research Football (GRF) demonstrate the scalability and promising performance of the framework.