Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraints

TL;DR

This paper explains why linear interpretability methods like probes and autoencoders succeed in transformer models, showing they are a consequence of architectural constraints that enforce invariant linear subspaces for semantic features.

Contribution

It introduces the Invariant Subspace Necessity theorem and the Self-Reference Property, providing a principled architectural explanation for linear interpretability in transformers.

Findings

Empirical validation across eight classification tasks confirms semantic alignment.

Theoretical framework unifies linear probes and autoencoders.

Demonstrates tokens provide geometric directions for features without labeled data.

Abstract

Linear probes and sparse autoencoders consistently recover meaningful structure from transformer representations -- yet why should such simple methods succeed in deep, nonlinear systems? We show this is not merely an empirical regularity but a consequence of architectural necessity: transformers communicate information through linear interfaces (attention OV circuits, unembedding matrices), and any semantic feature decoded through such an interface must occupy a context-invariant linear subspace. We formalize this as the \emph{Invariant Subspace Necessity} theorem and derive the \emph{Self-Reference Property}: tokens directly provide the geometric direction for their associated features, enabling zero-shot identification of semantic structure without labeled data or learned probes. Empirical validation in eight classification tasks and four model families confirms the alignment between class tokens and semantically related instances. Our framework provides \textbf{a principled architectural explanation} for why linear interpretability methods work, unifying linear probes and sparse autoencoders.