Learning Dynamics in RL Post-Training for Language Models

TL;DR

This paper analyzes the learning dynamics of reinforcement learning post-training in language models using an NTK framework, revealing how limited feature variability increases model confidence and reduces output diversity, and proposes a new classifier-first training strategy.

Contribution

It introduces an NTK-based analysis of RL post-training dynamics, explaining confidence increase and output diversity reduction, and proposes the CF-RL method to improve training efficiency.

Findings

RL updates increase model confidence due to limited feature variability

CF-RL accelerates training and enhances model confidence

The mechanism of CF-RL differs from supervised linear probing

Abstract

Reinforcement learning (RL) post-training is a critical stage in modern language model development, playing a key role in improving alignment and reasoning ability. However, several phenomena remain poorly understood, including the reduction in output diversity. To gain a broader understanding of RL post-training, we analyze the learning dynamics of RL post-training from a perspective that has been studied in supervised learning but remains underexplored in RL. We adopt an empirical neural tangent kernel (NTK) framework and decompose the NTK into two components to characterize how RL updates propagate across training samples. Our analysis reveals that limited variability in feature representations can cause RL updates to systematically increase model confidence, providing an explanation for the commonly observed reduction in output diversity after RL post-training. Furthermore, we show that effective learning in this regime depends on rapidly shaping the classifier, which directly affects the gradient component of the NTK. Motivated by these insights, we propose classifier-first reinforcement learning (CF-RL), a simple two-stage training strategy that prioritizes classifier updates before standard RL optimization. Experimental results validate our theoretical analysis by demonstrating increased model confidence and accelerated optimization under CF-RL. Additional analysis shows that the mechanism underlying CF-RL differs from that of linear-probing-then-fine-tuning in supervised learning. Overall, our study formalizes the learning dynamics of RL post-training and motivates further analysis and improvement.