UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation

TL;DR

UPS introduces a unified neural operator framework for PDE solving that leverages cross-modal adaptation and pretrained language models, achieving state-of-the-art results with significantly less data and compute.

Contribution

The paper proposes a novel cross-modal adaptation approach for building efficient, unified neural PDE solvers by warm-starting from pretrained LLMs and explicit alignment.

Findings

State-of-the-art performance on diverse PDE families.

Uses 4 times less data and 26 times less compute than previous models.

Capable of few-shot transfer to unseen PDEs.

Abstract

We present Unified PDE Solvers (UPS), a data- and compute-efficient approach to developing unified neural operators for diverse families of spatiotemporal PDEs from various domains, dimensions, and resolutions. UPS embeds different PDEs into a shared representation space and processes them using a FNO-transformer architecture. Rather than training the network from scratch, which is data-demanding and computationally expensive, we warm-start the transformer from pretrained LLMs and perform explicit alignment to reduce the modality gap while improving data and compute efficiency. The cross-modal UPS achieves state-of-the-art results on a wide range of 1D and 2D PDE families from PDEBench, outperforming existing unified models using 4 times less data and 26 times less compute. Meanwhile, it is capable of few-shot transfer to unseen PDE families and coefficients.