The Two-Stage Decision-Sampling Hypothesis: Understanding the Emergence of Self-Reflection in RL-Trained LLMs

TL;DR

This paper introduces the Two-Stage Decision-Sampling Hypothesis to explain how RL training enables self-reflection in large language models, emphasizing the roles of sampling and decision components in generating solutions and self-correction.

Contribution

It formalizes the Gradient Attribution Property and decomposes policy into sampling and decision stages, providing a mechanistic explanation for RL's success over supervised fine-tuning.

Findings

RL improves decision-making ($$) more than sampling ($$) in arithmetic reasoning.

Surrogate rewards exhibit Balanced Gradient Attribution, while SFT and KL penalties show Unbalanced Gradient Attribution.

Length-weighting creates asymmetric regularization constraining sampling, explaining RL's effectiveness.

Abstract

Self-reflection capabilities emerge in Large Language Models after RL post-training, with multi-turn RL achieving substantial gains over SFT counterparts. Yet the mechanism of how a unified optimization objective gives rise to functionally distinct capabilities of generating solutions and evaluating when to revise them remains opaque. To address this question, we introduce the Gradient Attribution Property to characterize how reward gradients distribute across policy components, formalized through the Two-Stage Decision-Sampling (DS) Hypothesis, which decomposes the policy into sampling ($\pi_{sample}$) for generation and decision ($\pi_{d}$) for verification. We prove that surrogate rewards exhibit Balanced Gradient Attribution, while SFT and KL penalties exhibit Unbalanced Gradient Attribution, with length-weighting creating asymmetric regularization that constrains $\pi_{sample}$ while leaving $\pi_{d}$ under-optimized, providing an theoretical explanation of why RL succeeds where SFT fails. We also empirically validate our theoretical predictions on arithmetic reasoning demonstrates that RL's superior generalization stems primarily from improved decision-making ($\pi_{d}$) rather than sampling capabilities, providing a first-principles mechanistic explanation for self-correction in thinking models.