Transolver is a Linear Transformer: Revisiting Physics-Attention through the Lens of Linear Attention

TL;DR

This paper reinterprets Physics-Attention as a form of linear attention, redesigns it into a more efficient Linear Attention Neural Operator, and demonstrates state-of-the-art results on PDE benchmarks with reduced complexity.

Contribution

It introduces a novel reformulation of Physics-Attention as linear attention and proposes LinearNO, achieving better performance with fewer parameters and lower computational cost.

Findings

State-of-the-art performance on six PDE benchmarks.

40% reduction in model parameters.

36% reduction in computational cost.

Abstract

Recent advances in Transformer-based Neural Operators have enabled significant progress in data-driven solvers for Partial Differential Equations (PDEs). Most current research has focused on reducing the quadratic complexity of attention to address the resulting low training and inference efficiency. Among these works, Transolver stands out as a representative method that introduces Physics-Attention to reduce computational costs. Physics-Attention projects grid points into slices for slice attention, then maps them back through deslicing. However, we observe that Physics-Attention can be reformulated as a special case of linear attention, and that the slice attention may even hurt the model performance. Based on these observations, we argue that its effectiveness primarily arises from the slice and deslice operations rather than interactions between slices. Building on this insight, we propose a two-step transformation to redesign Physics-Attention into a canonical linear attention, which we call Linear Attention Neural Operator (LinearNO). Our method achieves state-of-the-art performance on six standard PDE benchmarks, while reducing the number of parameters by an average of 40.0% and computational cost by 36.2%. Additionally, it delivers superior performance on two challenging, industrial-level datasets: AirfRANS and Shape-Net Car.