Multi-Agent Guided Policy Optimization
Yueheng Li, Guangming Xie, Zongqing Lu
TL;DR
MAGPO introduces a new multi-agent reinforcement learning framework that enhances centralized training with theoretical guarantees and scalable, coordinated exploration, outperforming existing methods across diverse tasks.
Contribution
It proposes MAGPO, a novel framework integrating centralized guidance with decentralized execution, providing theoretical guarantees and improved performance in multi-agent RL.
Findings
MAGPO outperforms strong CTDE baselines across 43 tasks.
MAGPO matches or surpasses fully centralized approaches.
Theoretical guarantees of monotonic policy improvement are established.
Abstract
Due to practical constraints such as partial observability and limited communication, Centralized Training with Decentralized Execution (CTDE) has become the dominant paradigm in cooperative Multi-Agent Reinforcement Learning (MARL). However, existing CTDE methods often underutilize centralized training or lack theoretical guarantees. We propose Multi-Agent Guided Policy Optimization (MAGPO), a novel framework that better leverages centralized training by integrating centralized guidance with decentralized execution. MAGPO uses an autoregressive joint policy for scalable, coordinated exploration and explicitly aligns it with decentralized policies to ensure deployability under partial observability. We provide theoretical guarantees of monotonic policy improvement and empirically evaluate MAGPO on 43 tasks across 6 diverse environments. Results show that MAGPO consistently outperforms…
Peer Reviews
Decision·ICLR 2026 Poster
Proposes a simple and conceptually elegant algorithmic framework for multi-agent RL. Demonstrates strong empirical performance across benchmark tasks.
It is not clearly explained how the guider (teacher) is reset to the learner (student) in practice. While in theory a product of decentralized policies defines an autoregressive joint policy, the mechanism of implementing this reset may be nontrivial depending on the network architecture.
The paper features a good discussion of related work in Sec. 2.2. The paper seems to do a good job motivating the research gap it aims to address. The experimental protocol seems solid, with the authors performing multiple runs and reporting confidence intervals. The experimental results seem reproducible. The experiments are also extensive, considering several different environments. The authors provide an ablation study. The work appears to feature a sufficient degree of novelty, even though I
I think the clarity of the paper needs to be further improved in some places. While the experimental results seem to support that the proposed method outperforms pervious baselines, I'm not sure whether the included baselines are well-representative of the previously proposed works; namely, I wonder if the authors are properly comparing their method against previously proposed *CTDS* methods for MARL (see my questions for additional information regarding this matter). This is my main point of co
The paper’s main strength lies in its conceptually coherent integration of centralized and decentralized learning in multi-agent reinforcement learning. By introducing the Multi-Agent Guided Policy Optimization (MAGPO) framework, it provides a principled way to leverage centralized guidance during training while maintaining decentralized deployability.
I have several concerns regarding this study, as summarized below. 1. The paper argues that the autoregressive guider policy alleviates the exponential growth of the joint action space. However, this sequential modeling still requires conditioning on all preceding agents’ actions and therefore may not scale well in the number of agents. The computational and communication costs of this approach under large agent populations are not analyzed, and all experiments are limited to small- or medium-s
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAuction Theory and Applications