Fundamental Limitations on Subquadratic Alternatives to Transformers

TL;DR

This paper proves that any subquadratic alternative to Transformers cannot perform certain important document similarity tasks, establishing a fundamental computational limitation based on complexity theory.

Contribution

It demonstrates that no subquadratic-time algorithm, including heuristic or alternative models, can match Transformers on document similarity tasks under a popular complexity conjecture.

Findings

Transformers can perform document similarity tasks efficiently.

Subquadratic algorithms cannot perform these tasks under the conjecture.

Any faster alternative to attention mechanisms cannot replicate Transformer performance.

Abstract

The Transformer architecture is widely deployed in many popular and impactful Large Language Models. At its core is the attention mechanism for calculating correlations between pairs of tokens. Performing an attention computation takes quadratic time in the input size, and had become the time bottleneck for transformer operations. In order to circumvent this, researchers have used a variety of approaches, including designing heuristic algorithms for performing attention computations faster, and proposing alternatives to the attention mechanism which can be computed more quickly. For instance, state space models such as Mamba were designed to replace attention with an almost linear time alternative. In this paper, we prove that any such approach cannot perform important tasks that Transformer is able to perform (assuming a popular conjecture from fine-grained complexity theory). We focus on document similarity tasks, where one is given as input many documents and would like to find a pair which is (approximately) the most similar. We prove that Transformer is able to perform this task, and we prove that this task cannot be performed in truly subquadratic time by any algorithm. Thus, any model which can be evaluated in subquadratic time - whether because of subquadratic-time heuristics for attention, faster attention replacements like Mamba, or any other reason - cannot perform this task. In other words, in order to perform tasks that (implicitly or explicitly) involve document similarity, one may as well use Transformer and cannot avoid its quadratic running time.