AOT-POT: Adaptive Operator Transformation for Large-Scale PDE Pre-training

TL;DR

AOT-POT introduces an adaptive transformation method for neural operators, enabling efficient multi-PDE pre-training by simplifying diverse solution operators into a unified, model-friendly form.

Contribution

The paper proposes an adaptive, input-dependent operator transformation technique that improves multi-PDE pre-training efficiency and accuracy over existing capacity-increasing methods.

Findings

Achieves state-of-the-art results on 12 PDE benchmarks with minimal additional parameters.

Reduces relative L2 error by up to 77.6%, averaging 40.9%.

Fine-tuning further decreases L2 error by up to 92% on unseen PDE types.

Abstract

Pre-training neural operators on diverse partial differential equation (PDE) datasets has emerged as a promising direction for building general-purpose surrogate models in scientific machine learning. However, the inherent complexity and structural diversity of PDE solution operators make multi-PDE pre-training fundamentally challenging. Existing methods mainly address this by increasing model capacity, while leaving the target solution operators unchanged. Inspired by classical numerical analysis, we instead propose to transform complex and diverse solution operators into simpler, better-aligned forms that are easier to model jointly. Since the optimal transformation varies across PDE types, it must be adaptive and input-dependent, allowing a single neural operator to approximate an entire family of operators. We instantiate this idea as AOT-POT (adaptive operator-transformation for pre-training operator transformer), which expands hidden representations into multiple parallel streams, adaptively aggregates and redistributes them before and after each sub-layer, and mixes streams through Sinkhorn-projected doubly stochastic matrices for stable training. These mechanisms together reshape diverse solution operators into a unified form that can be effectively modeled by a single architecture. Empirically, AOT-POT achieves state-of-the-art performance on 12 PDE benchmarks with only 3\% additional parameters, reducing relative L2 error by up to 77.6\% (40.9\% on average). Fine-tuning AOT-POT further reduces L2 error by up to 92\% on in-domain PDEs and 89\% on out-of-domain PDEs (unseen types during pre-training), demonstrating that adaptive operator transformation is an effective and complementary direction for advancing PDE foundation models beyond simply scaling model capacity.