Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learning

TL;DR

Tadpole is a foundation model for 3D PDEs that leverages autoencoders and online data generation to enable scalable, transferable, and multi-functional learning across diverse physical systems.

Contribution

It introduces a novel autoencoder-based foundation model trained on synthetic 3D PDE data with an efficient online framework, enabling transferability and multi-task capabilities.

Findings

Scales to hundreds of terabytes of training data without storage overhead.

Learns transferable representations across heterogeneous physical systems.

Achieves accurate downstream task performance with minimal fine-tuning.

Abstract

We introduce Tadpole, a novel foundation model for three-dimensional partial differential equations (PDEs) that addresses key challenges in transferability, scalability to high dimensionality, and multi-functionality. Tadpole is pre-trained as an autoencoder on synthetic 3D PDE data generated by an efficient online data-generation framework. This enables large-scale, diverse training without storage or I/O overhead, demonstrated by scaling to an equivalent of hundreds of terabytes of training data. By autoencoding single-channel spatial crops, Tadpole learns rich and transferable representations across heterogeneous physical systems with varying numbers of state variables and spatial resolutions. Although pre-trained solely as an autoencoder, Tadpole can be efficiently applied for multiple downstream tasks beyond reconstruction, including dynamics learning and generative modeling. For dynamics learning, we propose a novel parameter-efficient fine-tuning strategy that integrates low-rank adaptation, latent-space transformations, and reintroduced skip connections, achieving accurate temporal modeling with a minimal number of trainable parameters. Tadpole demonstrates strong fine-tuning performance across various downstream tasks, highlighting its versatility and effectiveness as a foundation model for 3D PDE learning. Source code and pre-trained weights of Tadpole are available at https://github.com/tum-pbs/tadpole