Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning

TL;DR

This paper introduces failure-prefix conditioning, a method to enhance reasoning models' learning from saturated problems by focusing on failure states, improving performance without needing harder problems.

Contribution

The paper proposes failure-prefix conditioning, a novel technique that leverages failure trajectories to improve learning from saturated reasoning problems in RLVR.

Findings

Failure-prefix conditioning improves model performance on saturated problems.

The method reduces performance degradation caused by misleading failure prefixes.

Iterative failure prefix refresh further enhances learning after initial plateaus.

Abstract

As Reinforcement Learning with Verifiable Rewards (RLVR) substantially improves the reasoning abilities of large language models (LLMs), a new bottleneck emerges: more training problems become saturated, that is, the LLM answers the questions correctly for nearly every rollout. On such problems, rewards provide little useful learning signal. While collecting harder problems is a natural response, it is costly and increasingly difficult. We propose failure-prefix conditioning, a simple method that unlocks the remaining signal in saturated problems by shifting exploration toward failure-prone reasoning states. By conditioning on prefixes of rare incorrect trajectories, the method improves the model's ability to recover from misleading early reasoning. We observe that failure-prefix conditioning consistently improves performance where standard RLVR stalls, and achieves gains comparable to training on newly collected medium-difficulty problems. We further analyze the model's robustness, finding that our method reduces performance degradation under misleading failure prefixes, albeit with a mild trade-off in adherence to correct early reasoning. Finally, we demonstrate that an iterative approach, which refreshes failure prefixes during training, unlocks additional gains after performance plateaus. Overall, our results show that saturated problems still contain valuable learning signal, and that failure-prefix conditioning provides an effective way to unlock it.