Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data

TL;DR

This paper introduces Constrained Uniform Top-K Sampling (CUTS), a novel decoding method that promotes diversity in model reasoning, preventing mode collapse and improving out-of-domain generalization in reinforcement learning for large language models.

Contribution

The paper proposes a new decoding strategy, CUTS, and a training framework, Mixed-CUTS, to enhance exploration and diversity, leading to better reasoning performance in large language models.

Findings

CUTS prevents policy degeneration and mode collapse.

Mixed-CUTS significantly improves Pass@1 accuracy on AIME25 by up to 15.1%.

Approach enhances out-of-domain generalization in RL for LLMs.

Abstract

Reinforcement Learning (RL) enhances LLM reasoning, yet a paradox emerges as models scale: strong base models saturate standard benchmarks (e.g., MATH), yielding correct but homogeneous solutions. In such environments, the lack of failure cases causes the advantage signal in group-relative algorithms (e.g., GRPO) to vanish, driving policies into mode collapse. To address this, we propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration. Unlike standard sampling that follows model biases, CUTS flattens the local optimization landscape by sampling uniformly from constrained high-confidence candidates. We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance. Experiments on Qwen3 models demonstrate that our approach prevents policy degeneration and significantly boosts out-of-domain generalization. Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO, validating that maintaining diversity within the semantic manifold is critical for rigorous reasoning.