Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection

TL;DR

This paper provides a theoretical and empirical analysis of reinforcement learning training dynamics in large language models, revealing how RL reshapes reasoning patterns and highlighting the importance of reward types and initial model quality.

Contribution

It offers a novel theoretical framework for understanding RL training in LLMs, focusing on pattern selection and the effects of different reward signals.

Findings

RL primarily optimizes critical tokens, reshaping reasoning patterns.

Model's initial reasoning quality influences convergence behavior.

Internal rewards can improve or degrade performance over training.

Abstract

While reinforcement learning (RL) demonstrated remarkable success in enhancing the reasoning capabilities of language models, the training dynamics of RL in LLMs remain unclear. In this work, we provide an explanation of the RL training process through empirical analysis and rigorous theoretical modeling. First, through systematic reasoning-pattern-level and token-level analysis across the RL training process, we show that while different reasoning patterns exhibit relatively stable success rates during training, RL primarily optimizes a sparse subset of critical tokens, thereby reshaping reasoning pattern distributions to affect model performance. Building on these empirical insights, we develop a theoretical framework to understand the training dynamics of RL with two typical rewards: verifiable reward (RLVR) and model's internal feedback (RLIF). For RLVR, we analyze the training dynamics under two special cases: one where models readily converge to optimal reasoning strategies, and another where optimization becomes challenging, revealing that the base model's reasoning quality is crucial for determining convergence behavior. For RLIF, we examine how internal rewards initially improve model performance but can potentially lead to degradation with continued training. Extensive experiments validate our findings, advancing both theoretical understanding and practical applications of RL in language model enhancement.