Replay-Based Continual Learning for Physics-Informed Neural Operators

TL;DR

This paper introduces a replay-based continual learning framework for physics-informed neural operators, enhancing their ability to adapt to out-of-distribution data without catastrophic forgetting.

Contribution

It proposes a simple, physics-informed replay strategy with distillation and transfer learning to improve continual learning in neural operators, validated on multiple physical problems.

Findings

Effectively mitigates catastrophic forgetting in neural operators.

Maintains fast adaptation to new out-of-distribution data.

Reduces training time and computational cost compared to joint training.

Abstract

Neural operators generally demonstrate strong predictive performance on in-distribution (ID) problems. However, a critical limitation of existing methods is their significant performance degradation when encountering out-of-distribution (OOD) data. To address this issue, this work introduces continual learning into physics-informed neural operators, with particular emphasis on neural operators built upon the Transolver architecture, and proposes a simple yet effective replay-based continual learning strategy. The proposed method is fully physics-informed and does not require labeled data, relying solely on input fields together with physical constraints for training. When new OOD data become available, a small number of past data are incorporated through a distillation-based constraint to preserve previously acquired knowledge and alleviate catastrophic forgetting. Meanwhile, a transfer learning LoRA is employed to enable rapid adaptation to the new data. The proposed framework is systematically validated on three representative physical problems, including the Darcy flow problem in fluid mechanics, a two-dimensional hyperelastic brain tumor problem in biomechanics, and a three-dimensional linear elastic Triply Periodic Minimal Surfaces problem in solid mechanics. The results demonstrate that the proposed method effectively mitigates catastrophic forgetting on previously learned data while maintaining fast adaptability to new data. Compared with conventional joint training strategies, the proposed method significantly improves training efficiency while reducing additional memory usage and computational cost.