Hindsight-Anchored Policy Optimization: Turning Failure into Feedback in Sparse Reward Settings

TL;DR

HAPO introduces a hindsight mechanism with a self-paced curriculum to improve reinforcement learning in sparse reward environments, ensuring unbiased policy updates and overcoming limitations of static teacher guidance.

Contribution

HAPO's synthetic success injection and gating mechanism provide a novel, theoretically grounded approach for stable, unbiased policy optimization in challenging sparse reward settings.

Findings

HAPO achieves asymptotic consistency and unbiased gradients.

The hindsight mechanism improves learning efficiency in sparse rewards.

The method surpasses static teacher forcing limitations.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for post-training reasoning models. However, group-based methods such as Group Relative Policy Optimization (GRPO) face a critical dilemma in sparse-reward settings: pure Reinforcement Learning (RL) suffers from advantage collapse and high-variance gradient estimation, while mixed-policy optimization introduces persistent distributional bias. To resolve this dilemma, we introduce Hindsight-Anchored Policy Optimization (HAPO). HAPO employs the Synthetic Success Injection (SSI) operator, a hindsight mechanism that selectively anchors optimization to teacher demonstrations during failure. This injection is governed by a Thompson sampling-inspired gating mechanism, creating an autonomous, self-paced curriculum. Theoretically, we demonstrate that HAPO achieves \textit{asymptotic consistency}: by naturally annealing the teacher signal as the policy improves, HAPO recovers the unbiased on-policy gradient. This ensures off-policy guidance acts as a temporary scaffold rather than a persistent ceiling, enabling the model to surpass the limitations of static teacher forcing.