Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning

TL;DR

This paper systematically analyzes how the entropy-performance trade-off in reinforcement learning with verifiable rewards affects large language model training, revealing stage-specific dynamics and proposing adaptive reward methods for improved learning.

Contribution

It provides a detailed empirical study of entropy-performance interactions at multiple granularities and introduces dynamic reward adjustment techniques based on these insights.

Findings

Entropy reduction in negative samples aids reasoning pattern learning.

High-entropy tokens at sequence ends correlate with learning efficiency.

Adaptive reward methods improve LLM training performance.

Abstract

Recently, reinforcement learning with verifiable rewards (RLVR) has been widely used for enhancing the reasoning abilities of large language models (LLMs). A core challenge in RLVR involves managing the exchange between entropy and performance of policies. Despite the importance of this exchange, a fine-grained understanding of when and how this exchange operates most effectively remains limited. To bridge this gap, we conduct a systematic empirical analysis of the entropy-performance exchange mechanism of RLVR across different levels of granularity. Specifically, we first divide the training process into two distinct stages based on entropy dynamics, i.e., rising stage and plateau stage, and then systematically investigate how this mechanism varies across stage-level, instance-level, and token-level granularitiess. Our analysis reveals that, in the rising stage, entropy reduction in negative samples facilitates the learning of effective reasoning patterns, which in turn drives rapid performance gains. Moreover, in the plateau stage, learning efficiency strongly correlates with high-entropy tokens present in low-perplexity samples and those located at the end of sequences. Motivated by these findings, we propose two methods that dynamically adjust the reward signal using perplexity and positional information to focus RL updates on tokens that exhibit high learning potential, achieving improvements compared to the baseline methods on various LLMs.