The Invisible Leash: Why RLVR May or May Not Escape Its Origin

TL;DR

This paper empirically investigates RLVR's effectiveness in enhancing LLM reasoning, revealing it often narrows exploration and fails to expand the model's reasoning horizon despite improving precision.

Contribution

It provides new empirical insights into RLVR's limitations, especially its tendency to restrict exploration and the entropy-reward trade-off affecting solution diversity.

Findings

RLVR improves pass@1 but often reduces empirical support.

RLVR increases token entropy but decreases answer diversity.

RLVR may overlook correct, underrepresented solutions.

Abstract

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising method for enhancing LLMs' capabilities. However, it remains unclear whether the current practice of RLVR truly expands a model's reasoning boundary or mainly amplifies high-reward outputs that the base model already knows, thereby improving precision. This study presents an empirical investigation that provides fresh insights into the limits of RLVR. We examine how RLVR can operate as a support-constrained optimization mechanism that may restrict the discovery of entirely original solutions, remaining constrained by the base model's initial distribution. We also identify an entropy-reward trade-off: while RLVR reliably enhances precision, it may progressively narrow exploration and potentially overlook correct yet underrepresented solutions. Extensive empirical experiments validate that while RLVR consistently improves \texttt{pass@1}, \textit{the shrinkage of empirical support generally outweighs the expansion of empirical support under larger sampling budgets}, failing to recover correct answers that were previously accessible to the base model. Interestingly, while RLVR sometimes increases token-level entropy, it results in greater uncertainty at each generation step and declining answer-level entropy. This indicates that these seemingly more uncertain paths ultimately converge onto a smaller set of distinct answers. Taken together, we reveal potential limits of RLVR in extending reasoning horizons. Breaking this invisible leash requires future innovations that seed probability mass into underrepresented solution regions.