Multi-Objective Loss Balancing for Physics-Informed Deep Learning

TL;DR

This paper introduces ReLoBRaLo, a novel self-adaptive loss balancing method for Physics-Informed Neural Networks that improves training accuracy and efficiency by effectively weighting multiple loss components.

Contribution

The paper proposes ReLoBRaLo, a new adaptive loss balancing scheme for PINNs, outperforming existing methods in accuracy and computational efficiency across multiple PDE benchmarks.

Findings

ReLoBRaLo outperforms existing loss scaling methods in accuracy.

ReLoBRaLo reduces computational overhead during training.

The method is effective on multiple benchmark PDE problems.

Abstract

Physics-Informed Neural Networks (PINN) are algorithms from deep learning leveraging physical laws by including partial differential equations together with a respective set of boundary and initial conditions as penalty terms into their loss function. In this work, we observe the significant role of correctly weighting the combination of multiple competitive loss functions for training PINNs effectively. To this end, we implement and evaluate different methods aiming at balancing the contributions of multiple terms of the PINNs loss function and their gradients. After reviewing of three existing loss scaling approaches (Learning Rate Annealing, GradNorm and SoftAdapt), we propose a novel self-adaptive loss balancing scheme for PINNs named \emph{ReLoBRaLo} (Relative Loss Balancing with Random Lookback). We extensively evaluate the performance of the aforementioned balancing schemes by solving both forward as well as inverse problems on three benchmark PDEs for PINNs: Burgers' equation, Kirchhoff's plate bending equation and Helmholtz's equation. The results show that ReLoBRaLo is able to consistently outperform the baseline of existing scaling methods in terms of accuracy, while also inducing significantly less computational overhead.