ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning

TL;DR

ADHint introduces an adaptive hinting framework for reinforcement learning that explicitly incorporates difficulty priors to improve learning stability, exploration, and generalization across diverse tasks.

Contribution

The paper presents ADHint, a novel RL method that integrates difficulty priors into hint scheduling and advantage estimation, enhancing stability and performance.

Findings

Outperforms existing hint-based RL methods in various domains.

Improves out-of-distribution generalization.

Achieves superior reasoning capabilities across modalities.

Abstract

To address the limited capability expansion and low sample efficiency of Reinforcement Learning (RL), recent methods have integrated ''hints'' into post-training, which are prefix segments of complete reasoning trajectories, aiming for powerful knowledge expansion and reasoning generalization. However, existing hint-based RL methods often neglect the role of difficulty in the hint-ratio schedule and relative-advantage estimation, resulting in unstable learning and excessive imitation of off-policy hints. To address this, we propose ADHint, which explicitly integrates difficulty into both processes to achieve a better trade-off between exploration and imitation. Specifically, we propose Adaptive Hint with Sample Difficulty Prior, which evaluates the difficulty of each sample under the current policy to schedule an appropriate hint ratio for rollout generation. Furthermore, we introduce Consistency-based Gradient Modulation alongside Selective Masking for Hint Preservation, which jointly modulate token-level gradients within hints to prevent biased and destructive updates. Additionally, we propose Advantage Estimation with Rollout Difficulty Posterior, which leverages the relative difficulty of rollouts with and without hints to compute their respective advantages, yielding more balanced updates. Extensive experiments across diverse modalities, model scales, model families, and domains demonstrate that ADHint achieves superior reasoning capabilities and out-of-distribution generalization. Code and datasets will be made publicly available upon paper acceptance.