Multi-scale Time-stepping of Partial Differential Equations with Transformers

TL;DR

This paper introduces a transformer-based neural network model with multi-scale hierarchical time-stepping to efficiently predict the evolution of PDEs, achieving comparable or superior accuracy to existing methods like FNO and other transformers.

Contribution

The work presents a novel transformer architecture combined with multi-scale time-stepping for PDE prediction, improving speed and accuracy over previous neural operators.

Findings

Achieves similar or better accuracy than Fourier Neural Operator.

Reduces prediction error over time with hierarchical time-stepping.

Demonstrates effectiveness on Navier-Stokes equations.

Abstract

Developing fast surrogates for Partial Differential Equations (PDEs) will accelerate design and optimization in almost all scientific and engineering applications. Neural networks have been receiving ever-increasing attention and demonstrated remarkable success in computational modeling of PDEs, however; their prediction accuracy is not at the level of full deployment. In this work, we utilize the transformer architecture, the backbone of numerous state-of-the-art AI models, to learn the dynamics of physical systems as the mixing of spatial patterns learned by a convolutional autoencoder. Moreover, we incorporate the idea of multi-scale hierarchical time-stepping to increase the prediction speed and decrease accumulated error over time. Our model achieves similar or better results in predicting the time-evolution of Navier-Stokes equations compared to the powerful Fourier Neural Operator (FNO) and two transformer-based neural operators OFormer and Galerkin Transformer.