On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR

TL;DR

This paper investigates how RLVR models focus on rank-1 components for reasoning, revealing implicit reward overfitting, low-rank dynamics, and spectral properties that influence training and continual learning.

Contribution

It uncovers the low-rank spectral dynamics and implicit reward overfitting phenomena in RLVR, providing insights into model parameter shaping and training behavior.

Findings

RLVR's reasoning is concentrated in rank-1 components.

Models can perform well with low rewards during training.

Singular value distributions in RLVR are heavy-tailed.

Abstract

Recent extensive research has demonstrated that the enhanced reasoning capabilities acquired by models through Reinforcement Learning with Verifiable Rewards (RLVR) are primarily concentrated within the rank-1 components. Predicated on this observation, we employed Periodic Rank-1 Substitution and identified a counterintuitive phenomenon: RLVR may exhibit implicit reward overfitting to the training dataset. Specifically, the model can achieve satisfactory performance on the test set even when its rewards remain relatively low during the training process. Furthermore, we characterize three distinct properties of RL training: (1) The effective rank-1 component in RLVR don't maintain other model knowledge except mathematical reasoning capability. (2) RLVR fundamentally functions by optimizing a specific singular spectrum. The distribution of singular values of almost all linear layers in RLVR-trained model behaves like heavy-tailed distribution. (3) the left singular vectors associated with rank-1 components demonstrate a stronger alignment tendency during training, which echoes the discovery that RLVR is optimizing sampling efficiency in essence. Taken together, our findings and analysis further reveal how RLVR shapes model parameters and offer potential insights for improving existing RL paradigms or other training paradigms to implement continual learning.