Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization

TL;DR

Graph-GRPO introduces a group-based policy optimization method for multi-agent topology learning, reducing reward noise and improving stability and performance in large language model systems.

Contribution

It proposes a novel group relative policy optimization framework that enhances topology learning by mitigating reward variance and enabling precise credit assignment.

Findings

Outperforms state-of-the-art baselines in reasoning and code generation tasks.

Achieves more stable training and better identification of critical communication links.

Effectively reduces reward noise from task difficulty variance.

Abstract

Optimizing communication topology is fundamental to the efficiency and effectiveness of Large Language Model (LLM)-based Multi-Agent Systems (MAS). While recent approaches utilize reinforcement learning to dynamically construct task-specific graphs, they typically rely on single-sample policy gradients with absolute rewards (e.g., binary correctness). This paradigm suffers from severe gradient variance and the credit assignment problem: simple queries yield non-informative positive rewards for suboptimal structures, while difficult queries often result in failures that provide no learning signal. To address these challenges, we propose Graph-GRPO, a novel topology optimization framework that integrates Group Relative Policy Optimization. Instead of evaluating a single topology in isolation, Graph-GRPO samples a group of diverse communication graphs for each query and computes the advantage of specific edges based on their relative performance within the group. By normalizing rewards across the sampled group, our method effectively mitigates the noise derived from task difficulty variance and enables fine-grained credit assignment. Extensive experiments on reasoning and code generation benchmarks demonstrate that Graph-GRPO significantly outperforms state-of-the-art baselines, achieving superior training stability and identifying critical communication pathways previously obscured by reward noise.