CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution

TL;DR

CoVerRL introduces a co-evolution framework where a generator and verifier iteratively improve each other to overcome the consensus trap in label-free reasoning, significantly enhancing reasoning accuracy and self-verification in large language models.

Contribution

The paper proposes a novel generator-verifier co-evolution approach that mitigates the consensus trap in label-free reinforcement learning for reasoning tasks.

Findings

Outperforms label-free baselines by 4.7-5.9% on reasoning benchmarks.

Self-verification accuracy improves from 55% to over 85%.

Demonstrates effective co-evolution of reasoning and verification capabilities.

Abstract

Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.