Mitigating Overthinking in Large Reasoning Models via Difficulty-aware Reinforcement Learning

TL;DR

This paper introduces DiPO, a reinforcement learning framework that makes large reasoning models aware of task difficulty, reducing unnecessary reasoning length and resource use without sacrificing performance.

Contribution

It proposes a novel difficulty-aware training method that models task complexity and adjusts reasoning behavior, addressing overthinking in large reasoning models.

Findings

Reduces reasoning length and resource consumption

Maintains reasoning performance while compressing thoughts

Effectively models task difficulty with minimal manual annotation

Abstract

Large Reasoning Models (LRMs) achieve explicit chain-of-thought expansion by imitating deep thinking behaviors of humans, demonstrating excellent performance in complex task scenarios. However, the deep-thinking mode often leads to unnecessarily lengthy reasoning and resource inefficiency when handling simple tasks. This overthinking phenomenon may arise from the generation preference triggered by the reward function during post-training. Existing research attempts to mitigate overthinking from the perspective of prompt design or model training, but generally underestimates the importance of task difficulty awareness, which makes it difficult for LRMs to effectively allocate reasoning resources. In this paper, we propose Difficulty-aware Policy Optimization (DiPO), a reinforcement learning-based LRM training framework. DiPO encourages LRM to spontaneously model task complexity, and integrates them into reinforcement learning framework to adjust the generation preferences introduced by post-training. A difficulty modeling method based on model self-reasoning is proposed, which significantly reduces the dependence on manual annotation and formalize task complexity. We further develop a difficulty-signal-enhanced reward function that incorporates a penalty for lengthy reasoning while considering reasoning performance and output format. Experimental results indicate that DiPO enables the model to spontaneously adjust inference overhead, significantly reducing redundant tokens without losing performance due to thought compression.