State Rank Dynamics in Linear Attention LLMs

TL;DR

This paper investigates the internal state dynamics of Linear Attention Large Language Models, revealing a fundamental spectral bifurcation in attention heads and proposing a pruning method to reduce computational overhead without significant accuracy loss.

Contribution

It uncovers the State Rank Stratification phenomenon in linear attention models and introduces a zero-shot pruning strategy based on these insights.

Findings

Spectral bifurcation among attention heads: low-rank and high-rank groups.

State rank properties are intrinsic and stable across contexts.

Pruning reduces KV-cache overhead by 38.9% with minimal accuracy impact.

Abstract

Linear Attention Large Language Models (LLMs) offer a compelling recurrent formulation that compresses context into a fixed-size state matrix, enabling constant-time inference. However, the internal dynamics of this compressed state remain largely opaque. In this work, we present a comprehensive study on the runtime state dynamics of state-of-the-art Linear Attention models. We uncover a fundamental phenomenon termed State Rank Stratification, characterized by a distinct spectral bifurcation among linear attention heads: while one group maintains an effective rank oscillating near zero, the other exhibits rapid growth that converges to an upper bound. Extensive experiments across diverse inference contexts reveal that these dynamics remain strikingly consistent, indicating that the identity of a head,whether low-rank or high-rank,is an intrinsic structural property acquired during pre-training, rather than a transient state dependent on the input data. Furthermore, our diagnostic probes reveal a surprising functional divergence: low-rank heads are indispensable for model reasoning, whereas high-rank heads exhibit significant redundancy. Leveraging this insight, we propose Joint Rank-Norm Pruning, a zero-shot strategy that achieves a 38.9\% reduction in KV-cache overhead while largely maintaining model accuracy.