Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models

TL;DR

This paper introduces PITA, a physics-informed self-supervised framework that improves auto-regressive PDE foundation models by aligning physical dynamics across time steps, enhancing accuracy and robustness especially on out-of-distribution data.

Contribution

The paper proposes a novel physics-informed temporal alignment method that enhances auto-regressive PDE models without relying on known physics priors, improving generalization.

Findings

PITA significantly improves model accuracy on PDE data.

PITA enhances robustness against out-of-distribution data.

The method generalizes well without known physics priors.

Abstract

Auto-regressive partial differential equation (PDE) foundation models have shown great potential in handling time-dependent data. However, these models suffer from the shortcut problem deeply rooted in auto-regressive prediction, causing error accumulation. The challenge becomes particularly evident for out-of-distribution data, as the pretraining performance may approach random model initialization for downstream tasks with long-term dynamics. To deal with this problem, we propose physics-informed temporal alignment (PITA), a self-supervised learning framework inspired by inverse problem solving. Specifically, PITA aligns the physical dynamics discovered at different time steps on each given PDE trajectory by integrating physics-informed constraints into the self-supervision signal. The alignment is derived from observation data without relying on known physics priors, indicating strong generalization ability to the out-of-distribution data. Extensive experiments show that PITA significantly enhances the accuracy and robustness of existing foundation models on diverse time-dependent PDE data. The code is available at https://github.com/SCAILab-USTC/PITA.