Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control

TL;DR

This paper introduces a theoretical framework for understanding entropy mechanics in reinforcement learning with verifiable rewards, revealing token-level polarity effects and proposing a new optimization method that improves exploration and exploitation balance.

Contribution

It develops the concept of entropy polarity at the token level, analyzes structural asymmetries in entropy regulation, and proposes PAPO, a novel entropy-aware policy optimization method.

Findings

Entropy polarity reliably predicts entropy changes during training.

Positive and negative polarity branches complement each other in exploration and exploitation.

PAPO outperforms baselines in mathematical reasoning and agentic benchmarks.

Abstract

Policy entropy has emerged as a fundamental measure for understanding and controlling exploration in reinforcement learning with verifiable rewards (RLVR) for LLMs. However, existing entropy-aware methods mainly regulate entropy through global objectives, while the token-level mechanism by which sampled policy updates reshape policy entropy remains underexplored. In this work, we develop a theoretical framework of entropy mechanics in RLVR. Our analysis yields a first-order approximation of the entropy change, giving rise to entropy polarity, a signed token-level quantity that predicts how much a sampled update expands or contracts entropy. This analysis further reveals a structural asymmetry: reinforcing frequent high-probability tokens triggers contraction tendencies, whereas expansive tendencies typically require lower-probability samples or stronger distributional correction. Empirically, we show that entropy polarity reliably predicts entropy changes, and that positive and negative polarity branches play complementary roles in preserving exploration while strengthening exploitation. Building on these insights, we propose Polarity-Aware Policy Optimization (PAPO), which preserves both polarity branches and implements entropy control through advantage reweighting. With the empirical entropy trajectory as an online phase signal, PAPO adaptively reallocates optimization pressure between entropy-expanding and entropy-contracting updates. Experiments on mathematical reasoning and agentic benchmarks show that PAPO consistently outperforms competitive baselines, while delivering superior training efficiency and substantial reward improvements.