Universal Approximation of Operators with Transformers and Neural Integral Operators

TL;DR

This paper proves that transformers and neural integral operators can universally approximate a wide class of operators between Banach spaces, extending their applicability in functional analysis.

Contribution

It establishes the universal approximation capabilities of transformers and neural integral operators for operators in Banach spaces, including generalized and modified versions.

Findings

Transformers are universal approximators of integral operators between Hölder spaces.

Neural integral operators based on Gavurin integral are universal for operators between Banach spaces.

Modified transformers using Leray-Schauder mappings are universal for operators between arbitrary Banach spaces.

Abstract

We study the universal approximation properties of transformers and neural integral operators for operators in Banach spaces. In particular, we show that the transformer architecture is a universal approximator of integral operators between H\"older spaces. Moreover, we show that a generalized version of neural integral operators, based on the Gavurin integral, are universal approximators of arbitrary operators between Banach spaces. Lastly, we show that a modified version of transformer, which uses Leray-Schauder mappings, is a universal approximator of operators between arbitrary Banach spaces.