MirrorLA: Reflecting Feature Map for Vision Linear Attention

TL;DR

MirrorLA introduces a geometric approach using learnable reflections to enhance linear attention in Transformers, significantly improving performance while maintaining linear computational complexity.

Contribution

It proposes MirrorLA, a novel framework that replaces passive truncation with active reorientation via Householder reflections, boosting information retention in linear attention models.

Findings

Achieves state-of-the-art results on standard benchmarks.

Restores representational density without sacrificing efficiency.

Effectively balances local discriminability and global covariance mixing.

Abstract

Linear attention significantly reduces the computational complexity of Transformers from quadratic to linear, yet it consistently lags behind softmax-based attention in performance. We identify the root cause of this degradation as the non-negativity constraint imposed on kernel feature maps: standard projections like ReLU act as "passive truncation" operators, indiscriminately discarding semantic information residing in the negative domain. We propose MirrorLA, a geometric framework that substitutes passive truncation with active reorientation. By leveraging learnable Householder reflections, MirrorLA rotates the feature geometry into the non-negative orthant to maximize information retention. Our approach restores representational density through a cohesive, multi-scale design: it first optimizes local discriminability via block-wise isometries, stabilizes long-context dynamics using variance-aware modulation to diversify activations, and finally, integrates dispersed subspaces via cross-head reflections to induce global covariance mixing. MirrorLA achieves state-of-the-art performance across standard benchmarks, demonstrating that strictly linear efficiency can be achieved without compromising representational fidelity.