On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

TL;DR

This paper emphasizes the importance of the direction of updates in RLVR for LLM reasoning, proposing methods that leverage this insight to enhance reasoning accuracy without additional training.

Contribution

It introduces the signed token-level log probability difference as a key metric and develops practical methods for test-time and training-time improvements based on update direction.

Findings

Direction of RLVR updates better identifies reasoning-critical changes

Amplifying policy along update direction improves reasoning accuracy

Focusing training on high-impact tokens enhances model performance

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the \textbf{magnitude} of these updates, largely overlooking their \textbf{direction}. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference $\Delta\log p$ between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that $\Delta\log p$ more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a \textit{test-time extrapolation} method that amplifies the policy along the learned $\Delta\log p$ direction to improve reasoning accuracy without further training; (2) a \textit{training-time reweighting} method that focuses learning on low-probability (corresponding to higher $\Delta\log p$) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.