Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectification

TL;DR

Anchored Policy Optimization (APO) introduces a support coverage approach to reinforcement learning, preventing collapse and improving diversity and accuracy in policy optimization by balancing sharpening and restorative forces.

Contribution

APO shifts from shape matching to support coverage, providing a gradient-aligned method that enhances support support and enables elastic recovery to prevent policy collapse.

Findings

APO significantly improves Pass@1 accuracy.

APO restores diversity lost in standard policy gradients.

APO effectively prevents Recursive Space Contraction.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is increasingly viewed as a tree pruning mechanism. However, we identify a systemic pathology termed Recursive Space Contraction (RSC), an irreversible collapse driven by the combined dynamics of positive sharpening and negative squeezing, where the sampling probability of valid alternatives vanishes. While Kullback-Leibler (KL) regularization aims to mitigate this, it imposes a rigid Shape Matching constraint that forces the policy to mimic the reference model's full density, creating a gradient conflict with the sharpening required for correctness. We propose Anchored Policy Optimization (APO), shifting the paradigm from global Shape Matching to Support Coverage. By defining a Safe Manifold based on the reference model's high-confidence support, APO permits aggressive sharpening for efficiency while selectively invoking a restorative force during error correction to prevent collapse. We theoretically derive that APO serves as a gradient-aligned mechanism to maximize support coverage, enabling an Elastic Recovery that re-inflates valid branches. Empirical evaluations on mathematical benchmarks demonstrate that APO breaks the accuracy-diversity trade-off, significantly improving Pass@1 while restoring the Pass@K diversity typically lost by standard policy gradient methods.