Linear Dynamics in the RLVR Training of Large Language Models

TL;DR

This paper reveals a consistent linear regime in RLVR training of large language models, enabling faster training and improved performance through extrapolation techniques.

Contribution

It uncovers the linear dynamics in RLVR training, providing theoretical insights and practical extrapolation methods to enhance training efficiency and model performance.

Findings

RLVR training exhibits a highly linear evolution of weights and log-probabilities.

Weight-space extrapolation achieves 6.1x training speedup with performance comparable to standard RL.

Output-space extrapolation outperforms standard RL, improving benchmark scores by 4.2%.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven significant performance gains in reasoning-oriented large language models (LLMs), yet its internal training dynamics remain largely a black box. In this work, we perform a comprehensive trajectory-level analysis of RLVR and uncover a striking regularity: across various model families, RL algorithms, and training configurations, RLVR consistently enters a robust linear regime, where both parameter weights and output log-probabilities, measured rigorously via teacher-forced evaluation, evolve in a highly linear manner ($R^2 > 0.7$). Through controlled experiments and theoretical analysis, we demonstrate that this linearity is not a coincidence, but stems from the high-variance, noisy nature of RLVR training signals, which act as a low-pass filter to concentrate optimization along a stable, low-dimensional drift. Moreover, we show that this linear structure is not merely descriptive but powerfully predictive and actionable. Specifically, weight-space extrapolation matches the performance of standard RL optimization while achieving a 6.1x training speedup through periodic re-grounding. Meanwhile, output-space extrapolation serves as a lightweight intervention that effectively bypasses late-stage model collapse, consistently outperforming standard RL across mathematical and coding benchmarks, with an average performance improvement of 4.2%. Our code is available at https://github.com/Miaow-Lab/RLVR-Linearity.