Targeted Exploration via Unified Entropy Control for Reinforcement Learning

TL;DR

The paper introduces UEC-RL, a unified framework for targeted exploration and stabilization in reinforcement learning, significantly improving reasoning capabilities in large models by maintaining exploration diversity and training stability.

Contribution

UEC-RL provides a novel targeted exploration and stabilization mechanism that enhances RL performance in large language and vision-language models.

Findings

UEC-RL achieves a 37.9% relative improvement over GRPO on Geometry3K.

Experiments show UEC-RL improves Pass@1 and Pass@$k$ metrics.

UEC-RL maintains stable training while expanding exploration.

Abstract

Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge prematurely and lose diversity. Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain optimization stability. We propose Unified Entropy Control for Reinforcement Learning (UEC-RL), a framework that provides targeted mechanisms for exploration and stabilization. UEC-RL activates more exploration on difficult prompts to search for potential and valuable reasoning trajectories. In parallel, a stabilizer prevents entropy from growing uncontrollably, thereby keeping training stable as the model consolidates reliable behaviors. Together, these components expand the search space when needed while maintaining robust optimization throughout training. Experiments on both LLM and VLM reasoning tasks show consistent gains over RL baselines on both Pass@1 and Pass@$k$. On Geometry3K, UEC-RL achieves a 37.9\% relative improvement over GRPO, indicating that it sustains effective exploration without compromising convergence and underscoring UEC-RL as a key for scaling RL-based reasoning in large models. Our code is available at https://github.com/597358816/UEC-RL.