Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs

TL;DR

This paper introduces iFNO, an incremental learning approach for Fourier Neural Operators that progressively increases frequency modes and resolution, leading to faster training and improved generalization in solving large-scale PDEs.

Contribution

The paper proposes the Incremental Fourier Neural Operator (iFNO), a novel method that incrementally increases frequency modes and resolution during training for efficient PDE solving.

Findings

iFNO reduces training time by 30%.

iFNO achieves 10% lower testing error.

iFNO uses 20% fewer frequency modes than standard FNO.

Abstract

Fourier Neural Operators (FNO) offer a principled approach to solving challenging partial differential equations (PDE) such as turbulent flows. At the core of FNO is a spectral layer that leverages a discretization-convergent representation in the Fourier domain, and learns weights over a fixed set of frequencies. However, training FNO presents two significant challenges, particularly in large-scale, high-resolution applications: (i) Computing Fourier transform on high-resolution inputs is computationally intensive but necessary since fine-scale details are needed for solving many PDEs, such as fluid flows, (ii) selecting the relevant set of frequencies in the spectral layers is challenging, and too many modes can lead to overfitting, while too few can lead to underfitting. To address these issues, we introduce the Incremental Fourier Neural Operator (iFNO), which progressively increases both the number of frequency modes used by the model as well as the resolution of the training data. We empirically show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets. Our method demonstrates a 10% lower testing error, using 20% fewer frequency modes compared to the existing Fourier Neural Operator, while also achieving a 30% faster training.