MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data

TL;DR

This paper introduces MaD-Scientist, a novel AI approach using massive PINN-based prior data and Transformer models to solve PDEs like convection-diffusion-reaction equations, demonstrating robustness and potential for low-cost scientific pre-training.

Contribution

It presents a new methodology combining physics-informed neural networks and Transformer architectures for zero-shot PDE solving with approximated prior data, advancing scientific foundation models.

Findings

Pre-training with approximated PINN data remains robust.

Transformer models effectively predict PDE solutions without knowledge of equations.

Low-cost data can be used for pre-training SFMs, reducing reliance on high-cost numerical data.

Abstract

Large language models (LLMs), like ChatGPT, have shown that even trained with noisy prior data, they can generalize effectively to new tasks through in-context learning (ICL) and pre-training techniques. Motivated by this, we explore whether a similar approach can be applied to scientific foundation models (SFMs). Our methodology is structured as follows: (i) we collect low-cost physics-informed neural network (PINN)-based approximated prior data in the form of solutions to partial differential equations (PDEs) constructed through an arbitrary linear combination of mathematical dictionaries; (ii) we utilize Transformer architectures with self and cross-attention mechanisms to predict PDE solutions without knowledge of the governing equations in a zero-shot setting; (iii) we provide experimental evidence on the one-dimensional convection-diffusion-reaction equation, which demonstrate that pre-training remains robust even with approximated prior data, with only marginal impacts on test accuracy. Notably, this finding opens the path to pre-training SFMs with realistic, low-cost data instead of (or in conjunction with) numerical high-cost data. These results support the conjecture that SFMs can improve in a manner similar to LLMs, where fully cleaning the vast set of sentences crawled from the Internet is nearly impossible.