Revisiting Entropy in Reinforcement Learning for Large Reasoning Models

TL;DR

This paper investigates entropy dynamics in reinforcement learning with verifiable rewards for large language models, identifying key factors influencing entropy collapse and proposing a novel reweighting method to mitigate it.

Contribution

It provides a comprehensive analysis of entropy behavior in RLVR for LLMs and introduces Positive-Advantage Reweighting to effectively control entropy during training.

Findings

Entropy collapse correlates with response diversity, calibration, and performance.

Clipping thresholds, update frequency, and data diversity significantly affect entropy.

Tokens with positive advantages are primary drivers of entropy collapse.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a prominent paradigm for enhancing the reasoning capabilities of large language models (LLMs). However, the entropy of LLMs usually collapses during RLVR training, leading to premature convergence to suboptimal local minima and hindering further performance improvement. Although various approaches have been proposed to mitigate entropy collapse, a comprehensive study of entropy in RLVR remains lacking. To bridge this gap, we conduct extensive experiments to investigate the entropy dynamics of LLMs trained with RLVR and analyze how model entropy correlates with response diversity, calibration, and performance across various benchmarks. Our results identify three key factors that influence entropy: the clipping thresholds in the optimization objective, the number of off-policy updates, and the diversity of the training data. Furthermore, through both theoretical analysis and empirical validation, we demonstrate that tokens with positive advantages are the primary drivers of entropy collapse. Motivated by this insight, we propose Positive-Advantage Reweighting, a simple yet effective approach that regulates model entropy by adjusting the loss weights assigned to tokens with positive advantages during RLVR training, while maintaining competitive performance.