Attention Mechanisms Through the Lens of Numerical Methods: Approximation Methods and Alternative Formulations

TL;DR

This survey reviews various approximation and reformulation techniques for attention mechanisms in Transformers, emphasizing numerical linear algebra tools to improve scalability and efficiency.

Contribution

It systematically classifies and unifies diverse fast attention methods within a numerical analysis framework, highlighting interdisciplinary opportunities.

Findings

Classifies attention approximation methods based on numerical principles

Discusses kernel-inspired reformulations and architectural variants

Highlights potential for further mathematical contributions to scalable attention

Abstract

The attention mechanism is the computational core of modern Transformer architectures, but its quadratic complexity in the input sequence length is the bottleneck for large-scale inference. This has motivated a rapidly growing body of work aimed at accelerating attention through approximation and reformulation. In this survey, we revisit attention mechanisms through the lens of numerical analysis, with a particular emphasis on tools and perspectives from numerical linear algebra. Our goal is twofold: first, we aim to systematically review and classify fast approximation methods according to the numerical principles they exploit. These include sparsity and clustering approaches, low-rank and subspace projection techniques, randomized sketching methods, and tensor-based decompositions. We also discuss kernel-inspired reformulations of attention and recent architectural variants, such as Latent Attention, that modify the standard softmax formulation to improve efficiency. Second, by presenting these developments within a unified mathematical framework, we aim to bridge the gap between disciplines and highlight opportunities for further contributions from computational mathematics, particularly numerical linear algebra, to the design of scalable attention mechanisms.