SchoenbAt: Rethinking Attention with Polynomial basis

TL;DR

SchoenbAt introduces a polynomial basis approach for kernelized attention, leveraging Schoenberg's theorem and random Maclaurin features to improve efficiency while maintaining accuracy in sequence modeling tasks.

Contribution

The paper proposes SchoenbAt, a novel polynomial basis-based kernelized attention method using Schoenberg's theorem, expanding beyond Fourier bases and providing theoretical and empirical validation.

Findings

SchoenbAt achieves faster computation compared to existing methods.

It maintains competitive accuracy in real-world datasets.

Theoretical analysis confirms unbiasedness and error bounds.

Abstract

Kernelized attention extends the attention mechanism by modeling sequence correlations through kernel functions, making significant progresses in optimizing attention. Under the guarantee of harmonic analysis theory, kernel functions can be expanded with basis functions, inspiring random feature-based approaches to enhance the efficiency of kernelized attention while maintaining predictive performance. However, current random feature-based works are limited to the Fourier basis expansions under Bochner's theorem. We propose Schoenberg's theorem-based attention (SchoenbAt), which approximates dot-product kernelized attention with the polynomial basis under Schoenberg's theorem via random Maclaurin features and applies a two-stage regularization to constrain the input space and restore the output scale, acting as a drop-in replacement of dot-product kernelized attention. Our theoretical proof of the unbiasedness and concentration error bound of SchoenbAt supports its efficiency and accuracy as a kernelized attention approximation, which is also empirically validated under various random feature dimensions. Evaluations on real-world datasets demonstrate that SchoenbAt significantly enhances computational speed while preserving competitive performance in terms of precision, outperforming several efficient attention methods.