SLAY: Geometry-Aware Spherical Linearized Attention with Yat-Kernel

TL;DR

SLAY introduces a geometry-aware, linear-time attention mechanism based on spherical Yat-kernels, achieving near-softmax performance with improved scalability and outperformance of prior linear attention methods.

Contribution

It presents a novel spherical Yat-kernel-based attention that is geometry-aware, positive definite, and computationally efficient, closely approximating softmax attention.

Findings

SLAY achieves near-softmax performance.

SLAY outperforms prior linear attention methods.

SLAY maintains linear time and memory complexity.

Abstract

We propose a new class of linear-time attention mechanisms based on a relaxed and computationally efficient formulation of the recently introduced E-Product, often referred to as the Yat-kernel (Bouhsine, 2025). The resulting interactions are geometry-aware and inspired by inverse-square interactions in physics. Our method, Spherical Linearized Attention with Yat Kernels (SLAY), constrains queries and keys to the unit sphere so that attention depends only on angular alignment. Using Bernstein's theorem, we express the spherical Yat-kernel as a nonnegative mixture of polynomial-exponential product kernels and derive a strictly positive random-feature approximation enabling linear-time O(L) attention. We establish positive definiteness and boundedness on the sphere and show that the estimator yields well-defined, nonnegative attention scores. Empirically, SLAY achieves performance that is nearly indistinguishable from standard softmax attention while retaining linear time and memory scaling, and consistently outperforms prior linear-time attention mechanisms such as Performers and Cosformers. To the best of our knowledge, SLAY represents the closest linear-time approximation to softmax attention reported to date, enabling scalable Transformers without the typical performance trade-offs of attention linearization.