HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignment

TL;DR

This paper introduces HEAL, a framework that improves exploration in few-shot RLVR by aligning entropy dynamics between target and general domains, leading to better reasoning with limited data.

Contribution

HEAL combines selective high-value data incorporation with entropy dynamics alignment to enhance exploration and reasoning in low-resource RLVR settings.

Findings

HEAL improves few-shot RLVR performance across multiple domains.

Using only 32 target samples, HEAL matches or surpasses models trained with 1K samples.

Entropy Dynamics Alignment effectively mitigates entropy collapse and promotes diverse exploration.

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has proven effective for training reasoning-oriented large language models, but existing methods largely assume high-resource settings with abundant training data. In low-resource scenarios, RLVR is prone to more severe entropy collapse, which substantially limits exploration and degrades reasoning performance. To address this issue, we propose Hybrid-domain Entropy dynamics ALignment (HEAL), a framework tailored for few-shot RLVR. HEAL first selectively incorporates high-value general-domain data to promote more diverse exploration. Then, we introduce Entropy Dynamics Alignment (EDA), a reward mechanism that aligns trajectory-level entropy dynamics between the target and general domains, capturing both entropy magnitude and fine-grained variation. Through this alignment, EDA not only further mitigates entropy collapse but also encourages the policy to acquire more diverse exploration behaviors from the general domain. Experiments across multiple domains show that HEAL consistently improves few-shot RLVR performance. Notably, using only 32 target-domain samples, HEAL matches or even surpasses full-shot RLVR trained with 1K target-domain samples.