HCPO: Hierarchical Conductor-Based Policy Optimization in Multi-Agent Reinforcement Learning

TL;DR

This paper introduces HCPO, a hierarchical, conductor-based policy optimization method for multi-agent reinforcement learning that improves exploration, coordination, and performance without requiring inter-agent communication during execution.

Contribution

The paper proposes a novel hierarchical conductor-based framework and algorithm that enhance joint policy expressiveness and coordination in MARL, with theoretical guarantees and practical benefits.

Findings

HCPO outperforms baseline methods on multiple benchmarks.

HCPO maintains centralized training benefits while eliminating inter-agent communication during execution.

Theoretical guarantees ensure monotonic policy improvement.

Abstract

In cooperative Multi-Agent Reinforcement Learning (MARL), efficient exploration is crucial for optimizing the performance of joint policy. However, existing methods often update joint policies via independent agent exploration, without coordination among agents, which inherently constrains the expressive capacity and exploration of joint policies. To address this issue, we propose a conductor-based joint policy framework that directly enhances the expressive capacity of joint policies and coordinates exploration. In addition, we develop a Hierarchical Conductor-based Policy Optimization (HCPO) algorithm that instructs policy updates for the conductor and agents in a direction aligned with performance improvement. A rigorous theoretical guarantee further establishes the monotonicity of the joint policy optimization process. By deploying local conductors, HCPO retains centralized training benefits while eliminating inter-agent communication during execution. Finally, we evaluate HCPO on three challenging benchmarks: StarCraftII Multi-agent Challenge, Multi-agent MuJoCo, and Multi-agent Particle Environment. The results indicate that HCPO outperforms competitive MARL baselines regarding cooperative efficiency and stability.