Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity

TL;DR

This paper demonstrates that transformer models can serve as universal neural operators for solving PDEs with finite regularity, outperforming DeepONet in accuracy across diverse dynamical systems.

Contribution

It establishes the universal approximation property of transformers as neural operators and applies them to PDE solutions with low regularity, a novel application in this context.

Findings

Transformers outperform DeepONet in accuracy for PDE solutions.

Transformers are capable of modeling solutions with finite regularity.

Transformers are more computationally expensive than DeepONet.

Abstract

Neural operator learning models have emerged as very effective surrogates in data-driven methods for partial differential equations (PDEs) across different applications from computational science and engineering. Such operator learning models not only predict particular instances of a physical or biological system in real-time but also forecast classes of solutions corresponding to a distribution of initial and boundary conditions or forcing terms. % DeepONet is the first neural operator model and has been tested extensively for a broad class of solutions, including Riemann problems. Transformers have not been used in that capacity, and specifically, they have not been tested for solutions of PDEs with low regularity. % In this work, we first establish the theoretical groundwork that transformers possess the universal approximation property as operator learning models. We then apply transformers to forecast solutions of diverse dynamical systems with solutions of finite regularity for a plurality of initial conditions and forcing terms. In particular, we consider three examples: the Izhikevich neuron model, the tempered fractional-order Leaky Integrate-and-Fire (LIF) model, and the one-dimensional Euler equation Riemann problem. For the latter problem, we also compare with variants of DeepONet, and we find that transformers outperform DeepONet in accuracy but they are computationally more expensive.