Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networks

TL;DR

This paper identifies a training failure mode in Physics Informed Neural Networks caused by scale imbalances and introduces an inverse-Dirichlet weighting strategy to improve training stability, accuracy, and robustness in multi-scale and sequential settings.

Contribution

The paper proposes the inverse-Dirichlet weighting method to address scale imbalance issues in PINNs, enhancing training reliability and performance.

Findings

Orders of magnitude improvement in accuracy and convergence for multi-scale turbulence models.

Inverse-Dirichlet weighting prevents catastrophic forgetting in sequential inverse modeling.

Effective in various applications, outperforming conventional PINN training.

Abstract

We characterize and remedy a failure mode that may arise from multi-scale dynamics with scale imbalances during training of deep neural networks, such as Physics Informed Neural Networks (PINNs). PINNs are popular machine-learning templates that allow for seamless integration of physical equation models with data. Their training amounts to solving an optimization problem over a weighted sum of data-fidelity and equation-fidelity objectives. Conflicts between objectives can arise from scale imbalances, heteroscedasticity in the data, stiffness of the physical equation, or from catastrophic interference during sequential training. We explain the training pathology arising from this and propose a simple yet effective inverse-Dirichlet weighting strategy to alleviate the issue. We compare with Sobolev training of neural networks, providing the baseline of analytically $\boldsymbol{\epsilon}$-optimal training. We demonstrate the effectiveness of inverse-Dirichlet weighting in various applications, including a multi-scale model of active turbulence, where we show orders of magnitude improvement in accuracy and convergence over conventional PINN training. For inverse modeling using sequential training, we find that inverse-Dirichlet weighting protects a PINN against catastrophic forgetting.