ETR: Outcome-Guided Elastic Trust Regions for Policy Optimization

TL;DR

This paper introduces Elastic Trust Regions (ETR), a dynamic policy optimization method that adapts constraints based on signal quality, leading to improved performance and sustained exploration in reinforcement learning tasks.

Contribution

The paper proposes ETR, a novel adaptive trust region mechanism that dynamically adjusts constraints according to advantage and variance, addressing limitations of static trust regions in policy optimization.

Findings

ETR outperforms GRPO on AIME and MATH benchmarks.

ETR maintains higher policy entropy during training.

ETR achieves superior accuracy in outcome-driven reinforcement learning.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an important paradigm for unlocking reasoning capabilities in large language models, exemplified by the success of OpenAI o1 and DeepSeek-R1. Currently, Group Relative Policy Optimization (GRPO) stands as the dominant algorithm in this domain due to its stable training and critic-free efficiency. However, we argue that GRPO suffers from a structural limitation: it imposes a uniform, static trust region constraint across all samples. This design implicitly assumes signal homogeneity, a premise misaligned with the heterogeneous nature of outcome-driven learning, where advantage magnitudes and variances fluctuate significantly. Consequently, static constraints fail to fully exploit high-quality signals while insufficiently suppressing noise, often precipitating rapid entropy collapse. To address this, we propose \textbf{E}lastic \textbf{T}rust \textbf{R}egions (\textbf{ETR}), a dynamic mechanism that aligns optimization constraints with signal quality. ETR constructs a signal-aware landscape through dual-level elasticity: at the micro level, it scales clipping boundaries based on advantage magnitude to accelerate learning from high-confidence paths; at the macro level, it leverages group variance to implicitly allocate larger update budgets to tasks in the optimal learning zone. Extensive experiments on AIME and MATH benchmarks demonstrate that ETR consistently outperforms GRPO, achieving superior accuracy while effectively mitigating policy entropy degradation to ensure sustained exploration.