The Routing and Filtering Structure of Attention

TL;DR

This paper analyzes the structure of attention in transformers, decomposing it into routing and filtering components, and introduces a diagnostic parameterization to improve interpretability and efficiency.

Contribution

It introduces $S$-$D$ attention to disentangle routing from filtering, revealing spectral cascades and enabling simplified, efficient attention mechanisms.

Findings

Routing operates at low rank, below the allocated capacity.

Linearizing early layers of $S$-$D$ attention costs less than 5% perplexity.

Cascade architectures reduce attention parameters significantly with minimal perplexity increase.

Abstract

The attention interaction matrix $QK^{\top}$ contains two entangled computations: a skew-symmetric component that redistributes information between positions (routing) and a symmetric component that scales mutual relevance (filtering). We decompose 1776 heads across five pretrained transformers and find routing operating at low rank, well below the routing capacity allocated by the weight kernel. We introduce $S$-$D$ attention as a diagnostic parameterization that disentangles routing from filtering by construction with guaranteed stability ($\mathrm{Re}(\lambda) \le 0$) and trains stably without layer normalization. When disentangled and unnormalized, routing self-organizes into a spectral cascade, effective rank $2$ at the first layer, expanding with depth across six scales from 7M to 355M parameters. The cascade predicts where attention can be simplified: linearizing the first seven layers of 125M $S$-$D$ attention costs ${<}5\%$ perplexity, whereas standard attention collapses under the same intervention. The linearizable region widens with depth. Replacing the first four layers with ELU+1 linear attention reaches within $1.4\%$ of baseline at full head dimension. Cascade-allocated architectures trade attention parameters for perplexity ($47\%-65\%$ fewer attention parameters at $+3.9\%$ to $+8.4\%$ PPL). The routing-filtering decomposition makes the spectral budget legible; the cascade makes it actionable.