Physics-Guided Transformer (PGT): Physics-Aware Attention Mechanism for PINNs

TL;DR

The paper introduces PGT, a physics-guided transformer that embeds physical laws directly into the attention mechanism, improving the reconstruction of physical fields from sparse data.

Contribution

It proposes a novel neural architecture that incorporates physical dynamics into attention, enhancing stability and accuracy in physics-informed learning.

Findings

PGT achieves low relative L2 error of 5.9e-3 on 1D heat equation.

PGT attains a PDE residual of 8.3e-4 in 2D Navier-Stokes simulations.

Outperforms traditional PINNs and sinusoidal methods in sparse data scenarios.

Abstract

Reconstructing continuous physical fields from sparse, irregular observations is a central challenge in scientific machine learning, particularly for systems governed by partial differential equations (PDEs). Existing physics-informed methods typically enforce governing equations as soft penalty terms during optimization, often leading to gradient imbalance, instability, and degraded physical consistency under limited data. We introduce the Physics-Guided Transformer (PGT), a neural architecture that embeds physical structure directly into the self-attention mechanism. Specifically, PGT incorporates a heat-kernel-derived additive bias into attention logits, encoding diffusion dynamics and temporal causality within the representation. Query coordinates attend to these physics-conditioned context tokens, and the resulting features are decoded using a FiLM-modulated sinusoidal implicit network that adaptively controls spectral response. We evaluate PGT on the one-dimensional heat equation and two-dimensional incompressible Navier-Stokes systems. In sparse 1D reconstruction with 100 observations, PGT achieves a relative L2 error of 5.9e-3, significantly outperforming both PINNs and sinusoidal representations. In the 2D cylinder wake problem, PGT uniquely achieves both low PDE residual (8.3e-4) and competitive relative error (0.034), outperforming methods that optimize only one objective. These results demonstrate that embedding physics within attention improves stability, generalization, and physical fidelity under data-scarce conditions.