New Skills or Sharper Primitives? A Probabilistic Perspective on the Emergence of Reasoning in RLVR

TL;DR

This paper presents a probabilistic framework to explain how reinforcement learning with verifiable rewards (RLVR) enhances reasoning in large language models by sharpening atomic step probabilities, leading to emergent complex capabilities.

Contribution

It introduces a new probabilistic perspective on RLVR, demonstrating that improving atomic step success rates enables models to develop complex reasoning skills from single-step training.

Findings

RLVR amplifies existing skills and explores new solution paths.

Composite success correlates strongly with atomic step probabilities.

RLVR can cause skill trade-offs to maximize overall reward.

Abstract

Whether Reinforcement Learning with Verifiable Rewards (RLVR) endows Large Language Models (LLMs) with new capabilities or merely elicits latent traces remains a central debate. In this work, we align with the former view, proposing a probabilistic framework where capability is defined by instance-level solvability. We hypothesize that the emergence of complex reasoning can be driven by sharpening atomic step probabilities, which enables models to overcome the exponential decay of success rates inherent in multi-step reasoning chains. Utilizing the Algebrarium framework, we train models exclusively on single-step operations and evaluate their performance on unseen multi-step tasks. Our empirical results confirm that: (1) RLVR incentivizes the exploration of previously inaccessible solution paths by amplifying the model's existing skills; (2) composite performance is strictly governed by the joint probability of atomic steps, evidenced by high Pearson correlation coefficients ($\rho \in [0.69, 0.96]$); and (3) RLVR, acting as a global optimizer, can cause specific skills to be sacrificed to maximize aggregate reward. Our work offers a novel explanation for emergent abilities in RLVR, suggesting that the iterative optimization of solvable problems enables models to develop the capabilities to tackle previously unsolvable scenarios.