Universal Approximation Theorem for a Single-Layer Transformer

TL;DR

This paper establishes a universal approximation theorem for single-layer Transformers, demonstrating their capacity to approximate any continuous sequence-to-sequence function, thereby advancing theoretical understanding of these models.

Contribution

It provides the first formal proof that a single-layer Transformer with self-attention and ReLU can approximate any continuous sequence-to-sequence mapping.

Findings

Proves single-layer Transformers are universal approximators.

Provides formal mathematical foundation for Transformer capabilities.

Demonstrates practical implications through case studies.

Abstract

Deep learning employs multi-layer neural networks trained via the backpropagation algorithm. This approach has achieved success across many domains and relies on adaptive gradient methods such as the Adam optimizer. Sequence modeling evolved from recurrent neural networks to attention-based models, culminating in the Transformer architecture. Transformers have achieved state-of-the-art performance in natural language processing (for example, BERT and GPT-3) and have been applied in computer vision and computational biology. However, theoretical understanding of these models remains limited. In this paper, we examine the mathematical foundations of deep learning and Transformers and present a novel theoretical result. We review key concepts from linear algebra, probability, and optimization that underpin deep learning, and we analyze the multi-head self-attention mechanism and the backpropagation algorithm in detail. Our main contribution is a universal approximation theorem for Transformers: we prove that a single-layer Transformer, comprising one self-attention layer followed by a position-wise feed-forward network with ReLU activation, can approximate any continuous sequence-to-sequence mapping on a compact domain to arbitrary precision. We provide a formal statement and a complete proof. Finally, we present case studies that demonstrate the practical implications of this result. Our findings advance the theoretical understanding of Transformer models and help bridge the gap between theory and practice.