In-Context Multi-Operator Learning with DeepOSets

TL;DR

This paper introduces DeepOSets, a simple yet powerful neural network architecture for multi-operator learning in scientific computing, capable of efficiently learning and generalizing multiple operators with theoretical guarantees.

Contribution

It provides a mathematical formulation for multi-operator learning, modifies DeepOSets for this task, and proves their universality, demonstrating fast training and strong generalization in experiments.

Findings

DeepOSets can learn multiple differential operators accurately.

The architecture achieves training times of minutes.

It generalizes to unseen operators and equations.

Abstract

An important application of neural networks to scientific computing has been the learning of non-linear operators. In this framework, a neural network is trained to fit a non-linear map between two infinite dimensional spaces, for example, the solution operator of ordinary and partial differential equations. Recently, inspired by the discovery of in-context learning for large language models, an even more ambitious paradigm has been explored, called multi-operator learning. In this approach, a neural network is trained to learn many different operators at the same time. In order to evaluate one of the learned operators, the network is passed example inputs and outputs to disambiguate the desired operator. In this work, we provide a precise mathematical formulation of the multi-operator learning problem. In addition, we modify a simple efficient architecture, called DeepOSets, for multi-operator learning and prove its universality for multi-operator learning. Finally, we provide a comprehensive set of experiments that demonstrate the ability of DeepOSets to learn multiple operators corresponding to different initial-value and boundary-value differential equations and use in-context examples to predict accurately the solutions corresponding to queries and differential equations not seen during training. The main advantage of DeepOSets is its architectural simplicity, which allows the derivation of theoretical guarantees and training times that are in the order of minutes, in contrast to similar transformer-based alternatives that are empirically justified and require hours of training.