Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models

TL;DR

This paper introduces a counterfactual intervention framework to evaluate the faithfulness of intermediate reasoning drafts in large reasoning models, revealing current models often lack faithful alignment with their reasoning processes.

Contribution

It proposes a systematic method to measure and analyze the faithfulness of thinking drafts in large reasoning models, addressing a key challenge in interpretability and reliability.

Findings

LRMs show selective faithfulness to reasoning steps

Models often fail to align draft conclusions with reasoning processes

The framework enables rigorous evaluation of reasoning faithfulness

Abstract

Large Reasoning Models (LRMs) have significantly enhanced their capabilities in complex problem-solving by introducing a thinking draft that enables multi-path Chain-of-Thought explorations before producing final answers. Ensuring the faithfulness of these intermediate reasoning processes is crucial for reliable monitoring, interpretation, and effective control. In this paper, we propose a systematic counterfactual intervention framework to rigorously evaluate thinking draft faithfulness. Our approach focuses on two complementary dimensions: (1) Intra-Draft Faithfulness, which assesses whether individual reasoning steps causally influence subsequent steps and the final draft conclusion through counterfactual step insertions; and (2) Draft-to-Answer Faithfulness, which evaluates whether final answers are logically consistent with and dependent on the thinking draft, by perturbing the draft's concluding logic. We conduct extensive experiments across six state-of-the-art LRMs. Our findings show that current LRMs demonstrate selective faithfulness to intermediate reasoning steps and frequently fail to faithfully align with the draft conclusions. These results underscore the need for more faithful and interpretable reasoning in advanced LRMs.