A unified framework for establishing the universal approximation of transformer-type architectures

TL;DR

This paper develops a unified theoretical framework to establish the universal approximation property of transformer architectures, including various attention mechanisms, by identifying key conditions and simplifying verification methods.

Contribution

It introduces a general sufficient condition for UAP in transformer models, extending prior residual network results to attention-based architectures with a simplified verification process.

Findings

Proves UAP for transformers with kernel-based attention mechanisms

Generalizes previous UAP results to new transformer architectures

Provides a foundation for designing transformers with guaranteed approximation capabilities

Abstract

We investigate the universal approximation property (UAP) of transformer-type architectures, providing a unified theoretical framework that extends prior results on residual networks to models incorporating attention mechanisms. Our work identifies token distinguishability as a fundamental requirement for UAP and introduces a general sufficient condition that applies to a broad class of architectures. Leveraging an analyticity assumption on the attention layer, we can significantly simplify the verification of this condition, providing a non-constructive approach in establishing UAP for such architectures. We demonstrate the applicability of our framework by proving UAP for transformers with various attention mechanisms, including kernel-based and sparse attention mechanisms. The corollaries of our results either generalize prior works or establish UAP for architectures not previously covered. Furthermore, our framework offers a principled foundation for designing novel transformer architectures with inherent UAP guarantees, including those with specific functional symmetries. We propose examples to illustrate these insights.