Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization

TL;DR

This paper analyzes entropy collapse in RLVR for language models, introduces a token-level entropy flow perspective, and proposes OPEFO to balance entropy dynamics, improving stability and performance.

Contribution

It offers a novel token-level entropy flow analysis and introduces OPEFO, an adaptive on-policy method to prevent entropy collapse in RLVR.

Findings

OPEFO stabilizes training across six reasoning benchmarks.

OPEFO enhances final performance compared to existing methods.

Entropy flow imbalance causes entropy collapse in RLVR.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning ability of large language models. However, widely used RLVR algorithms, such as GRPO, often suffer from entropy collapse, leading to premature determinism and unstable optimization. Existing remedies, including entropy regularization and ratio-based clipping heuristics, either control entropy in a coarse-grained manner or rely on approximate on-policy training. In this paper, we revisit entropy collapse from a token-level entropy flow perspective. Our analysis reveals that entropy-decreasing tokens consistently outweigh entropy-increasing ones, resulting in a severely imbalanced entropy flow. This perspective provides a unified explanation of entropy collapse in existing RLVR algorithms and highlights the importance of balancing entropy dynamics. Motivated by this analysis, we propose On-Policy Entropy Flow Optimization (OPEFO), an adaptive entropy flow balancing mechanism that rescales entropy-increasing and entropy-decreasing updates according to their contributions to entropy change, while remaining strict on-policy. Experiments on six mathematical reasoning benchmarks demonstrate that OPEFO improves training stability and final performance. We will release the code and models upon publication.