Counterfactual Credit Policy Optimization for Multi-Agent Collaboration

TL;DR

This paper introduces CCPO, a novel method for multi-agent reinforcement learning that improves credit assignment by estimating individual contributions through counterfactual trajectories, enhancing collaboration among large language models.

Contribution

CCPO is a new framework that assigns agent-specific learning signals using counterfactual baselines, improving credit assignment and collaboration in multi-agent LLM systems.

Findings

CCPO outperforms existing multi-agent RL baselines on reasoning benchmarks.

It reduces free-riding and enhances policy optimization.

The method is effective across different collaboration topologies.

Abstract

Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles and aggregating diverse hypotheses. Yet, reinforcement learning (RL) for such systems is often undermined by credit assignment: a shared global reward obscures individual contributions, inflating update variance and encouraging free-riding. We introduce Counterfactual Credit Policy Optimization (CCPO), a framework that assigns agent-specific learning signals by estimating each agent's marginal contribution through counterfactual trajectories. CCPO builds dynamic counterfactual baselines that simulate outcomes with an agent's contribution removed, yielding role-sensitive advantages for policy optimization. To further improve stability under heterogeneous tasks and data distributions, we propose a global-history-aware normalization scheme that calibrates advantages using global rollout statistics. We evaluate CCPO on two collaboration topologies: a sequential Think--Reason dyad and multi-agent voting. Across mathematical and logical reasoning benchmarks, CCPO mitigates free-riding and outperforms strong multi-agent RL baselines, yielding finer-grained and more effective credit assignment for collaborative LLM training. Our code is available at https://github.com/bhai114/ccpo.