Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective

TL;DR

This paper introduces a mechanistic approach to detect and mitigate reasoning hallucinations in large reasoning models, proposing new metrics, detection frameworks, and reinforcement learning techniques to improve factual accuracy and reasoning depth.

Contribution

It presents the Reasoning Score for quantifying reasoning depth, a detection framework for hallucinations, and an enhanced reinforcement learning algorithm for mitigation, advancing understanding and control of reasoning errors in LRMs.

Findings

Reasoning Score effectively distinguishes deep reasoning from shallow patterns.

The RHD framework achieves state-of-the-art hallucination detection performance.

GRPO-R reduces hallucination rates and improves reasoning quality.

Abstract

Large Reasoning Models (LRMs) have shown impressive capabilities in multi-step reasoning tasks. However, alongside these successes, a more deceptive form of model error has emerged--Reasoning Hallucination--where logically coherent but factually incorrect reasoning traces lead to persuasive yet faulty conclusions. Unlike traditional hallucinations, these errors are embedded within structured reasoning, making them more difficult to detect and potentially more harmful. In this work, we investigate reasoning hallucinations from a mechanistic perspective. We propose the Reasoning Score, which quantifies the depth of reasoning by measuring the divergence between logits obtained from projecting late layers of LRMs to the vocabulary space, effectively distinguishing shallow pattern-matching from genuine deep reasoning. Using this score, we conduct an in-depth analysis on the ReTruthQA dataset and identify two key reasoning hallucination patterns: early-stage fluctuation in reasoning depth and incorrect backtracking to flawed prior steps. These insights motivate our Reasoning Hallucination Detection (RHD) framework, which achieves state-of-the-art performance across multiple domains. To mitigate reasoning hallucinations, we further introduce GRPO-R, an enhanced reinforcement learning algorithm that incorporates step-level deep reasoning rewards via potential-based shaping. Our theoretical analysis establishes stronger generalization guarantees, and experiments demonstrate improved reasoning quality and reduced hallucination rates.