Fine-Tune Language Models as Multi-Modal Differential Equation Solvers

TL;DR

This paper introduces a multi-modal approach to in-context operator learning by integrating human knowledge via natural language descriptions, improving performance and reducing data needs in differential equation solving.

Contribution

It transforms in-context operator learning into a multi-modal framework using natural language, and fine-tunes language models to incorporate human insights for better differential equation solving.

Findings

Outperforms single-modal baselines in operator learning tasks

Enhances performance with multi-modal learning

Reduces data requirements for differential equation solving

Abstract

In the growing domain of scientific machine learning, in-context operator learning has shown notable potential in building foundation models, as in this framework the model is trained to learn operators and solve differential equations using prompted data, during the inference stage without weight updates. However, the current model's overdependence on function data overlooks the invaluable human insight into the operator. To address this, we present a transformation of in-context operator learning into a multi-modal paradigm. In particular, we take inspiration from the recent success of large language models, and propose using "captions" to integrate human knowledge about the operator, expressed through natural language descriptions and equations. Also, we introduce a novel approach to train a language-model-like architecture, or directly fine-tune existing language models, for in-context operator learning. We beat the baseline on single-modal learning tasks, and also demonstrated the effectiveness of multi-modal learning in enhancing performance and reducing function data requirements. The proposed method not only significantly enhanced the development of the in-context operator learning paradigm, but also created a new path for the application of language models.