An Attention-based Spatio-Temporal Neural Operator for Evolving Physics

TL;DR

This paper introduces ASNO, an attention-based neural operator that models evolving physical processes across space and time, improving prediction accuracy and interpretability in scientific machine learning tasks involving changing environments.

Contribution

The paper presents a novel neural architecture combining attention mechanisms and neural operators for better modeling of spatio-temporal physical processes with environmental adaptability.

Findings

ASNO outperforms existing models on SciML benchmarks.

It enhances interpretability by isolating contributions of states and external forces.

Demonstrates potential for physics discovery and engineering applications.

Abstract

In scientific machine learning (SciML), a key challenge is learning unknown, evolving physical processes and making predictions across spatio-temporal scales. For example, in real-world manufacturing problems like additive manufacturing, users adjust known machine settings while unknown environmental parameters simultaneously fluctuate. To make reliable predictions, it is desired for a model to not only capture long-range spatio-temporal interactions from data but also adapt to new and unknown environments; traditional machine learning models excel at the first task but often lack physical interpretability and struggle to generalize under varying environmental conditions. To tackle these challenges, we propose the Attention-based Spatio-Temporal Neural Operator (ASNO), a novel architecture that combines separable attention mechanisms for spatial and temporal interactions and adapts to unseen physical parameters. Inspired by the backward differentiation formula (BDF), ASNO learns a transformer for temporal prediction and extrapolation and an attention-based neural operator for handling varying external loads, enhancing interpretability by isolating historical state contributions and external forces, enabling the discovery of underlying physical laws and generalizability to unseen physical environments. Empirical results on SciML benchmarks demonstrate that ASNO outperforms over existing models, establishing its potential for engineering applications, physics discovery, and interpretable machine learning.