Heterogeneous Agent Collaborative Reinforcement Learning

TL;DR

HACRL introduces a new collaborative reinforcement learning framework for heterogeneous agents that improves efficiency and knowledge sharing without requiring coordinated deployment.

Contribution

The paper proposes HACRL and HACPO, enabling heterogeneous agents to share verified rollouts and mutually improve through principled, unbiased advantage estimation mechanisms.

Findings

HACPO outperforms GSPO with double rollouts by 3.6% on average.

HACRL enables independent inference while sharing training rollouts.

The methods improve all participating agents across diverse benchmarks.

Abstract

We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.