Improved Operator Learning by Orthogonal Attention

TL;DR

This paper introduces an orthogonal attention mechanism for neural operators, leveraging eigendecomposition to improve generalization and reduce overfitting in PDE solution learning, demonstrating superior performance on benchmark datasets.

Contribution

The paper proposes a novel orthogonal attention method based on eigendecomposition, providing a natural regularization to enhance neural operator training.

Findings

Outperforms baseline models on six benchmark datasets

Reduces overfitting through orthogonalization

Improves generalization across regular and irregular geometries

Abstract

Neural operators, as an efficient surrogate model for learning the solutions of PDEs, have received extensive attention in the field of scientific machine learning. Among them, attention-based neural operators have become one of the mainstreams in related research. However, existing approaches overfit the limited training data due to the considerable number of parameters in the attention mechanism. To address this, we develop an orthogonal attention based on the eigendecomposition of the kernel integral operator and the neural approximation of eigenfunctions. The orthogonalization naturally poses a proper regularization effect on the resulting neural operator, which aids in resisting overfitting and boosting generalization. Experiments on six standard neural operator benchmark datasets comprising both regular and irregular geometries show that our method can outperform competing baselines with decent margins.