Low-rank adaptive physics-informed HyperDeepONets for solving differential equations

TL;DR

This paper introduces PI-LoRA-HyperDeepONets, a low-rank, physics-informed neural network architecture that significantly reduces parameters and enhances accuracy in solving differential equations.

Contribution

It proposes a novel low-rank adaptation method for HyperDeepONets, improving efficiency and generalization in physics-informed operator learning.

Findings

Achieves up to 70% reduction in trainable parameters.

Consistently outperforms regular HyperDeepONets in accuracy.

Effective on both ordinary and partial differential equations.

Abstract

HyperDeepONets were introduced in Lee, Cho and Hwang [ICLR, 2023] as an alternative architecture for operator learning, in which a hypernetwork generates the weights for the trunk net of a DeepONet. While this improves expressivity, it incurs high memory and computational costs due to the large number of output parameters required. In this work we introduce, in the physics-informed machine learning setting, a variation, PI-LoRA-HyperDeepONets, which leverage low-rank adaptation (LoRA) to reduce complexity by decomposing the hypernetwork's output layer weight matrix into two smaller low-rank matrices. This reduces the number of trainable parameters while introducing an extra regularization of the trunk networks' weights. Through extensive experiments on both ordinary and partial differential equations we show that PI-LoRA-HyperDeepONets achieve up to 70\% reduction in parameters and consistently outperform regular HyperDeepONets in terms of predictive accuracy and generalization.