APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards

TL;DR

This paper introduces APR, a reward method that reduces structural redundancy in large reasoning models by penalizing repetitive verification after the reasoning process stabilizes, improving efficiency and performance.

Contribution

It formalizes the concept of the Reasoning Anchor and proposes a novel reward shaping technique to mitigate overthinking in large reasoning models.

Findings

APR achieves a better performance-efficiency trade-off on multiple datasets.

Models trained with APR require fewer computational resources.

APR maintains or improves reasoning accuracy while reducing unnecessary verification.

Abstract

Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs) but introduces a critical side-effect known as Overthinking. We conduct a preliminary study to rethink this phenomenon from a fine-grained perspective. We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process. We formally define this specific position where the answer first stabilizes as the Reasoning Anchor. By analyzing pre- and post-anchor reasoning behaviors, we uncover the structural redundancy fixed in LRMs: the meaningless repetitive verification after deriving the first complete answer, which we term the Answer-Stable Tail (AST). Motivated by this observation, we propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST. Leveraging the policy optimization algorithm suitable for length penalties, our APR models achieved the performance-efficiency Pareto frontier at 1.5B and 7B scales averaged across five mathematical reasoning datasets while requiring substantially fewer computational resources for RL training.