Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?

TL;DR

This paper demonstrates that a single-layer, low-rank self-attention Transformer can universally approximate continuous permutation-equivariant functions, resolving previous limitations related to depth requirements for data memorization.

Contribution

It establishes that one-layer, low-rank self-attention Transformers are universal approximators, clarifying their expressive power and addressing prior depth-related limitations.

Findings

One-layer self-attention with low-rank weights can memorize entire sequences.

Single-head Transformers can memorize finite samples.

Transformers with one self-attention layer and two feed-forward networks are universal approximators.

Abstract

Existing analyses of the expressive capacity of Transformer models have required excessively deep layers for data memorization, leading to a discrepancy with the Transformers actually used in practice. This is primarily due to the interpretation of the softmax function as an approximation of the hardmax function. By clarifying the connection between the softmax function and the Boltzmann operator, we prove that a single layer of self-attention with low-rank weight matrices possesses the capability to perfectly capture the context of an entire input sequence. As a consequence, we show that one-layer and single-head Transformers have a memorization capacity for finite samples, and that Transformers consisting of one self-attention layer with two feed-forward neural networks are universal approximators for continuous permutation equivariant functions on a compact domain.