Difficulty-Estimated Policy Optimization

TL;DR

This paper introduces DEPO, a new method that improves reasoning model training efficiency by dynamically estimating sample difficulty, reducing computational costs while maintaining performance.

Contribution

DEPO is the first framework to incorporate online difficulty estimation to optimize training data selection in reasoning models, enhancing efficiency and robustness.

Findings

DEPO reduces rollout costs by up to 2x.

DEPO maintains model performance despite reduced computation.

Empirical results validate DEPO's efficiency and robustness.

Abstract

Recent advancements in Large Reasoning Models (LRMs), exemplified by DeepSeek-R1, have underscored the potential of scaling inference-time compute through Group Relative Policy Optimization (GRPO). However, GRPO frequently suffers from gradient signal attenuation when encountering problems that are either too trivial or overly complex. In these scenarios, the disappearance of inter-group advantages makes the gradient signal susceptible to noise, thereby jeopardizing convergence stability. While variants like DAPO attempt to rectify gradient vanishing, they do not alleviate the substantial computational overhead incurred by exhaustive rollouts on low-utility samples. In this paper, we propose Difficulty-Estimated Policy Optimization (DEPO), a novel framework designed to optimize the efficiency and robustness of reasoning alignment. DEPO integrates an online Difficulty Estimator that dynamically assesses and filters training data before the rollout phase. This mechanism ensures that computational resources are prioritized for samples with high learning potential. Empirical results demonstrate that DEPO achieves up to a 2x reduction in rollout costs without compromising model performance. Our approach significantly lowers the computational barrier for training high-performance reasoning models, offering a more sustainable path for reasoning scaling. Code and data will be released upon acceptance.