LeMON: Learning to Learn Multi-Operator Networks

TL;DR

This paper introduces LeMON, a multi-operator learning framework for PDEs that leverages pretraining, fine-tuning, and meta-learning to efficiently adapt to new operators with limited data, outperforming single-operator models.

Contribution

It proposes a novel pretraining and fine-tuning strategy for multi-operator PDE learning, including zero-shot prediction and PDE-agnostic meta-learning, enhancing adaptability and efficiency.

Findings

Pretrained multi-operator models outperform single-operator models.

Models can predict unseen operators with limited fine-tuning data.

Low-rank adaptation reduces computational costs while maintaining accuracy.

Abstract

Single-operator learning involves training a deep neural network to learn a specific operator, whereas recent work in multi-operator learning uses an operator embedding structure to train a single neural network on data from multiple operators. Thus, multi-operator learning is capable of predicting a range of operators within one model. In this work, we propose pretraining and fine-tuning strategies for solving PDEs using multi-operator learning. One key aspect is that by increasing the number of families of operators used in pretraining, a PDE foundation model can be fine-tuned to downstream tasks involving new PDEs with a limited number of samples, thus outperforming single operator neural networks. Specifically, a multi-operator learning model pre-trained with data from diverse PDE families can predict unseen operators after fine-tuning with only a limited number of operators from the new family, enabling them to serve as a data-free PDE solver. We also show that the proposed training and fine-tuning method is able to predict new operators in zero-shot prediction without samples. Additionally, we introduce a PDE-agnostic meta-learning algorithm to improve the adaptability of the model to various PDEs by providing a better parameter initialization process. To address the needs of applications with limited computing resources, we explore low-rank adaptation methods that reduce computational costs while enhancing solver accuracy. Lastly, by examining the scaling law with respect to the number of operator families, we establish and highlight its potential for broad adaptation in PDE-solving tasks.