Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers

TL;DR

Conv-Basis introduces a convolution-based approximation for self-attention in transformers, significantly reducing computational complexity and enabling efficient processing of longer sequences in large language models.

Contribution

It develops a novel convolution basis system to approximate attention matrices, achieving near-linear time inference and training in transformers.

Findings

Achieves $O(knd \, \log n)$ inference time using FFT

Provides theoretical guarantees on runtime and approximation error

Demonstrates preliminary effectiveness in experiments

Abstract

The self-attention mechanism is the key to the success of transformers in recent Large Language Models (LLMs). However, the quadratic computational cost $O(n^2)$ in the input sequence length $n$ is a notorious obstacle for further improvement and scalability in longer contexts. In this work, we leverage the convolution-like structure of attention matrices to develop an efficient approximation method for attention computation using convolution matrices. We propose a $\mathsf{conv}$ basis system, analogous to the rank basis, and show that any lower triangular matrix can always be decomposed as a sum of structured convolution matrices in this basis. We then design a fast algorithm to approximate the attention matrix via a sum of such $k$ convolution matrices. This allows us to compute the attention {\it inference} via Fast Fourier Transforms (FFT) in $O(knd \log n)$ time, where $d$ is the hidden dimension, and thus achieve almost linear time $n^{1+o(1)}$ in the practical scenario where $kd = n^{o(1)}$. Furthermore, the attention {\it training forward} and {\it backward gradient} can be computed in $n^{1+o(1)}$ as well. We provide theoretical guarantees on the run time and approximation error and conduct preliminary experiments to evaluate its effectiveness. We hope our new paradigm for accelerating attention computation in transformer models can help their application to longer contexts.