Function graph transformers universally approximate operators between function spaces

TL;DR

This paper introduces function graph transformers, a measure-theoretic framework that universally approximates nonlinear operators between function spaces, enhancing understanding and capabilities of transformer-based operator learning.

Contribution

It develops a measure-theoretic approach to model operators with transformers, introduces function graph transformers, and proves their universal approximation capabilities.

Findings

Function graph transformers can approximate broad classes of nonlinear operators.

The framework accommodates regularized negative-order Sobolev inputs and multi-domain queries.

Universal approximation is achieved through compositions of softmax self-attention layers and MLPs.

Abstract

We study the approximation of nonlinear operators between function spaces by transformers. Our approach is to lift functions to measures supported on their graphs and leverage a recently introduced measure-theoretic view of transformers. A function $h$ is represented by its graph measure $\gamma_h$, with finite tokens $\{(x_j,h(x_j))\}_{j=1}^N$ being its empirical approximations. We show that this framework elegantly models discretization refinement via convergence of measures and provides a natural setting for operator learning. Within this framework, we introduce function graph transformers, a graph-preserving subclass of measure-theoretic transformers that maps graph measures to graph measures, which is to say that outputs remain single-valued functions. Crucially, this additional structure does not reduce generality: we prove that the resulting graph-preserving maps can be approximated by finite compositions of standard softmax self-attention layers and pointwise MLPs, yielding universal approximation results for broad classes of nonlinear operators. Unlike existing theoretical approaches to operator learning with transformers, the measure-theoretic framework also accommodates regularized negative-order Sobolev inputs for which discretization invariance is particularly challenging, as well as query points on different output domains. Overall, function graph transformers provide a continuum viewpoint and mathematical toolkit for transformer-based operator learning, clarifying the roles of positional encodings, graph structure, regularization, and ensuring consistency across discretizations.