Self-Hinting Language Models Enhance Reinforcement Learning

TL;DR

This paper introduces SAGE, a reinforcement learning framework that uses privileged hints during training to improve language model alignment under sparse rewards, leading to better performance without hints at test time.

Contribution

SAGE is a novel on-policy RL method that injects privileged hints during training to prevent advantage collapse and enhance diversity, outperforming previous methods like GRPO.

Findings

SAGE outperforms GRPO on 6 benchmarks across 3 LLMs.

Diverse self-hints serve as an adaptive curriculum for better learning.

SAGE achieves average improvements of +2.0, +1.2, and +1.3 points on different models.

Abstract

Group Relative Policy Optimization (GRPO) has recently emerged as a practical recipe for aligning large language models with verifiable objectives. However, under sparse terminal rewards, GRPO often stalls because rollouts within a group frequently receive identical rewards, causing relative advantages to collapse and updates to vanish. We propose self-hint aligned GRPO with privileged supervision (SAGE), an on-policy reinforcement learning framework that injects privileged hints during training to reshape the rollout distribution under the same terminal verifier reward. For each prompt $x$, the model samples a compact hint $h$ (e.g., a plan or decomposition) and then generates a solution $\tau$ conditioned on $(x,h)$. Crucially, the task reward $R(x,\tau)$ is unchanged; hints only increase within-group outcome diversity under finite sampling, preventing GRPO advantages from collapsing under sparse rewards. At test time, we set $h=\varnothing$ and deploy the no-hint policy without any privileged information. Moreover, sampling diverse self-hints serves as an adaptive curriculum that tracks the learner's bottlenecks more effectively than fixed hints from an initial policy or a stronger external model. Experiments over 6 benchmarks with 3 LLMs show that SAGE consistently outperforms GRPO, on average +2.0 on Llama-3.2-3B-Instruct, +1.2 on Qwen2.5-7B-Instruct and +1.3 on Qwen3-4B-Instruct. The code is available at https://github.com/BaohaoLiao/SAGE.