Linear Self-Attention Approximation via Trainable Feedforward Kernel

TL;DR

This paper introduces a trainable kernel-based method to approximate self-attention in Transformers, aiming to achieve linear complexity while maintaining high accuracy, thus improving computational efficiency.

Contribution

It extends kernelized attention methods by proposing a trainable kernel approach to better approximate self-attention with reduced complexity.

Findings

Achieves near-linear attention computation

Maintains high accuracy compared to standard attention

Demonstrates efficiency on benchmark tasks

Abstract

In pursuit of faster computation, Efficient Transformers demonstrate an impressive variety of approaches -- models attaining sub-quadratic attention complexity can utilize a notion of sparsity or a low-rank approximation of inputs to reduce the number of attended keys; other ways to reduce complexity include locality-sensitive hashing, key pooling, additional memory to store information in compacted or hybridization with other architectures, such as CNN. Often based on a strong mathematical basis, kernelized approaches allow for the approximation of attention with linear complexity while retaining high accuracy. Therefore, in the present paper, we aim to expand the idea of trainable kernel methods to approximate the self-attention mechanism of the Transformer architecture.