Compressible Softmax-Attended Language under Incompressible Attention

TL;DR

This paper analyzes the spectral properties of softmax attention in transformer language models, showing that most of the interaction energy is captured by a small number of components, indicating high compressibility.

Contribution

It decomposes the attention logit field into learned and generated parts and quantifies their spectral spectra across multiple models, revealing the data-driven nature of compressibility.

Findings

90% of logit variance captured by 2-11 singular components

Learned interaction matrix requires 38-75 components for similar variance

Spectral gap indicates high effective rank disparity

Abstract

Softmax attention defines an interaction through $d_h$ head dimensions, but not all dimensions carry equal weight once real text passes through. We decompose the attention logit field into a learned component and a generated component and measure their spectra separately. For all 5,888 KV heads in five transformer language models (124M--7B parameters, four architecture families), the logit energy field $\tilde{E}$ reaches 90\% of its variance in 2--11 singular components. The learned interaction matrix $W_Q^\mathrm{T} W_K$ needs 38--75 components for the same threshold out of $d_h \in {64, 128}$. The spectral gap is 5--25$\times$ in effective rank. The compressibility of softmax-attended language is a property of the data, not the frame that analyzes it.