Mapping Faithful Reasoning in Language Models

TL;DR

This paper introduces Concept Walk, a method to analyze internal reasoning processes of language models in activation space, helping distinguish between genuine and decorative reasoning traces for better oversight.

Contribution

It presents a novel framework, Concept Walk, for tracing internal concept dynamics in language models, enhancing understanding of reasoning faithfulness.

Findings

In easy cases, reasoning traces are ignored, indicating decorative reasoning.

In hard cases, internal activations show sustained shifts, indicating faithful reasoning.

Concept Walk helps identify when reasoning traces can be trusted.

Abstract

Chain-of-thought (CoT) traces promise transparency for reasoning language models, but prior work shows they are not always faithful reflections of internal computation. This raises challenges for oversight: practitioners may misinterpret decorative reasoning as genuine. We introduce Concept Walk, a general framework for tracing how a model's internal stance evolves with respect to a concept direction during reasoning. Unlike surface text, Concept Walk operates in activation space, projecting each reasoning step onto the concept direction learned from contrastive data. This allows us to observe whether reasoning traces shape outcomes or are discarded. As a case study, we apply Concept Walk to the domain of Safety using Qwen 3-4B. We find that in 'easy' cases, perturbed CoTs are quickly ignored, indicating decorative reasoning, whereas in 'hard' cases, perturbations induce sustained shifts in internal activations, consistent with faithful reasoning. The contribution is methodological: Concept Walk provides a lens to re-examine faithfulness through concept-specific internal dynamics, helping identify when reasoning traces can be trusted and when they risk misleading practitioners.