MSPT: Efficient Large-Scale Physical Modeling via Parallelized Multi-Scale Attention

TL;DR

MSPT is a scalable neural architecture that efficiently models large-scale physical systems by combining local and global attention mechanisms, enabling high accuracy with reduced computational resources.

Contribution

Introduces MSPT, a novel multi-scale attention-based neural network that efficiently handles millions of spatial elements in physics simulations using patch-based and global attention.

Findings

Achieves state-of-the-art accuracy on PDE benchmarks.

Scales to millions of points on a single GPU.

Reduces memory and computational costs significantly.

Abstract

A key scalability challenge in neural solvers for industrial-scale physics simulations is efficiently capturing both fine-grained local interactions and long-range global dependencies across millions of spatial elements. We introduce the Multi-Scale Patch Transformer (MSPT), an architecture that combines local point attention within patches with global attention to coarse patch-level representations. To partition the input domain into spatially-coherent patches, we employ ball trees, which handle irregular geometries efficiently. This dual-scale design enables MSPT to scale to millions of points on a single GPU. We validate our method on standard PDE benchmarks (elasticity, plasticity, fluid dynamics, porous flow) and large-scale aerodynamic datasets (ShapeNet-Car, Ahmed-ML), achieving state-of-the-art accuracy with substantially lower memory footprint and computational cost.