Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics

TL;DR

This paper investigates the limitations of spectral diagnostics in analyzing attention mechanisms in large language models, revealing orientation-blindness and proposing a two-axis diagnostic for better failure mode detection.

Contribution

It proves that symmetric spectral diagnostics cannot detect information flow direction and introduces a polarity prediction framework for attention failure modes.

Findings

Spectral diagnostics are orientation-blind and cannot distinguish operator transpose.

Transport capacity has a lower bound of 1/5, with window attention surpassing this floor.

Transport features retain interpretability up to 8B parameters, with polarity reversing as predicted.

Abstract

Large language models hallucinate in predictable ways: attention routing fails by over-concentrating on a narrow set of positions, or by spreading so diffusely that relevance is diluted, and the shape of the failure carries diagnostic signal. A widely used family of spectral methods analyzes the symmetric component of the degree-normalized attention operator, which governs transport capacity; we prove that every transpose-invariant spectral diagnostic of this operator is structurally orientation-blind (it cannot distinguish an operator from its transpose, and therefore cannot detect information-flow direction), with a quantitative converse establishing the asymmetry coefficient $G$ as the unique control parameter for direction. Pairing this with a closed-form bipartite-Cheeger landscape for canonical causal architectures, we show that uniform causal attention satisfies an $n$-independent floor $\phi \ge 1/5$ with worst cut at $t^\ast/n \approx 0.32$, while window attention pierces the floor as $O(w/n)$; failure modes are shape-different, not just value-different. The resulting two-axis diagnostic ($\phi$ for capacity, $G$ for direction) yields a falsifiable polarity prediction: bottleneck- and diffuse-dominated benchmarks should exhibit opposite polarity. Under length-controlled evaluation, transport features retain interpretable signal (LC-AUROC from 0.62 to 0.84) on tested models up to 8B parameters, with polarity reversing as predicted between HaluEval and MedHallu.