CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

TL;DR

CPMobius introduces a collaborative Coach-Player framework for data-free reinforcement learning that enhances reasoning abilities in large language models without external data, outperforming existing unsupervised methods.

Contribution

It proposes a novel cooperative Coach-Player paradigm inspired by human sports collaboration, enabling data-free training of reasoning models with improved performance.

Findings

Achieves +4.9 overall accuracy improvement on Qwen2.5-Math-7B-Instruct.

Outperforms existing unsupervised approaches like RENT and R-zero.

Demonstrates effective reasoning skill enhancement without external data.

Abstract

Large Language Models (LLMs) have demonstrated strong potential in complex reasoning, yet their progress remains fundamentally constrained by reliance on massive high-quality human-curated tasks and labels, either through supervised fine-tuning (SFT) or reinforcement learning (RL) on reasoning-specific data. This dependence renders supervision-heavy training paradigms increasingly unsustainable, with signs of diminishing scalability already evident in practice. To overcome this limitation, we introduce CPM\"obius (CPMobius), a collaborative Coach-Player paradigm for data-free reinforcement learning of reasoning models. Unlike traditional adversarial self-play, CPM\"obius, inspired by real world human sports collaboration and multi-agent collaboration, treats the Coach and Player as independent but cooperative roles. The Coach proposes instructions targeted at the Player's capability and receives rewards based on changes in the Player's performance, while the Player is rewarded for solving the increasingly instructive tasks generated by the Coach. This cooperative optimization loop is designed to directly enhance the Player's mathematical reasoning ability. Remarkably, CPM\"obius achieves substantial improvement without relying on any external training data, outperforming existing unsupervised approaches. For example, on Qwen2.5-Math-7B-Instruct, our method improves accuracy by an overall average of +4.9 and an out-of-distribution average of +5.4, exceeding RENT by +1.5 on overall accuracy and R-zero by +4.2 on OOD accuracy. Our codebase has been released at https://github.com/thunlp/CPMobius.