When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains

TL;DR

This paper introduces the Multi-Scale Attention Transformer ( extbackslash msat{}) for solving PDEs, demonstrating superior accuracy and efficiency over existing methods on complex geometries and providing insights into architecture selection and regularization effects.

Contribution

The paper presents extbackslash msat{}, a novel transformer-based architecture for PDE solving, with comprehensive empirical evaluation and theoretical analysis guiding architecture choice and regularization.

Findings

extbackslash msat{} achieves state-of-the-art accuracy on complex geometry PDE problems.

extbackslash msat{} significantly reduces inference time compared to Mamba-NO.

Physics regularization improves performance on diffusion problems but harms chaotic regimes.

Abstract

We study the problem of \emph{architecture selection} for deep learning models trained to solve partial differential equations (PDEs), asking when transformer-based architectures with learned attention outperform Fourier-domain neural operators. We introduce the \textbf{Multi-Scale Attention Transformer} (\msat{}), a deep learning architecture that encodes spatiotemporal solution histories as token sequences and trains end-to-end via a composite supervised objective with optional physics-informed regularization terms. We conduct a comprehensive empirical evaluation against nine baselines -- including physics-informed neural networks (PINNs), neural operators (FNO, DeepONet, GNOT), and state-space models (Mamba-NO) -- across five benchmark problems from the PINNacle suite, using identical train/test splits and reference data for all methods. \msat{} achieves state-of-the-art generalization on complex geometry problems ($L^2_\mathrm{rel} = 0.0101$ on Heat2D-CG, a $3.7\times$ improvement over FNO) at $34\,\mathrm{s}$ total inference vs.\ $120{,}812\,\mathrm{s}$ for Mamba-NO. Ablation studies over the physics regularization component reveal a precise inductive bias tradeoff: physics priors reduce test error on diffusion-dominated problems but degrade generalization on chaotic and recirculating-flow regimes, directly characterizing the prior misspecification boundary. Approximation error bounds as a function of domain boundary complexity $\kappa$ provide a theoretical basis for these empirical findings and a principled rule for architecture selection.