GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning

TL;DR

GHPO introduces an adaptive, difficulty-aware reinforcement learning framework that improves training stability and performance of large language models in complex reasoning tasks by dynamically balancing imitation and exploration strategies.

Contribution

The paper presents GHPO, a novel adaptive guidance method that calibrates task difficulty for stable and efficient LLM reinforcement learning, outperforming existing methods.

Findings

Achieves ~5% performance improvement on six math benchmarks

Enhances training stability and reasoning accuracy

Balances imitation and exploration for curriculum learning

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a powerful paradigm for facilitating the self-improvement of large language models (LLMs), particularly in the domain of complex reasoning tasks. However, prevailing on-policy RL methods often contend with significant training instability and inefficiency. This is primarily due to a capacity-difficulty mismatch, where the complexity of training data frequently outpaces the model's current capabilities, leading to critically sparse reward signals and stalled learning progress. This challenge is particularly acute for smaller, more resource-efficient LLMs. To overcome this, we introduce the Guided Hybrid Policy Optimization (GHPO), a novel difficulty-aware reinforcement learning framework. GHPO dynamically calibrates task difficulty by employing adaptive prompt refinement to provide targeted guidance. This unique approach adaptively balances direct imitation learning for problems currently beyond the model's reach with exploration-based reinforcement learning for more manageable tasks, effectively creating a smooth and optimized learning curriculum. Extensive experiments demonstrate that GHPO achieves an average performance gain of approximately 5% across six challenging mathematics benchmarks, consistently outperforming strong on-policy reinforcement learning and curriculum learning baselines. Further analysis confirms that our framework significantly enhances both training stability and final reasoning performance, thus offering a scalable and efficient solution for developing powerful and robust reasoning models.