Do Sparse Autoencoders Identify Reasoning Features in Language Models?

TL;DR

This paper investigates whether sparse autoencoders reliably identify reasoning features in language models, revealing that many features are sensitive to token interventions and emphasizing the importance of falsification in attribution.

Contribution

It introduces a falsification-based evaluation framework for probing reasoning features in language models using sparse autoencoders, highlighting their limitations.

Findings

Many contrastively selected features are highly sensitive to token interventions (45%-90%).

LLM-guided falsification can produce inputs that suppress reasoning trace activation.

Sparse autoencoders often capture low-dimensional correlates rather than true reasoning features.

Abstract

We study how reliably sparse autoencoders (SAEs) support claims about reasoning-related internal features in large language models. We first give a stylized analysis showing that sparsity-regularized decoding can preferentially retain stable low-dimensional correlates while suppressing high-dimensional within-behavior variation, motivating the possibility that contrastively selected "reasoning" features may concentrate on cue-like structure when such cues are coupled with reasoning traces. Building on this perspective, we propose a falsification-based evaluation framework that combines causal token injection with LLM-guided counterexample construction. Across 22 configurations spanning multiple model families, layers, and reasoning datasets, we find that many contrastively selected candidates are highly sensitive to token-level interventions, with 45%-90% activating after injecting only a few associated tokens into non-reasoning text. For the remaining context-dependent candidates, LLM-guided falsification produces targeted non-reasoning inputs that trigger activation and meaning-preserving paraphrases of top-activating reasoning traces that suppress it. A small steering study yields minimal changes on the evaluated benchmarks. Overall, our results suggest that, in the settings we study, sparse decompositions can favor low-dimensional correlates that co-occur with reasoning, underscoring the need for falsification when attributing high-level behaviors to individual SAE features. Code is available at https://github.com/GeorgeMLP/reasoning-probing.