Importance of hyper-parameter optimization during training of physics-informed deep learning networks

TL;DR

This paper demonstrates that hyper-parameter optimization is crucial when training physics-informed deep learning models, especially when using physics-based regularization, as it significantly impacts model accuracy and convergence.

Contribution

It shows that fine-tuning hyper-parameters for each physics-based regularization method improves model accuracy and convergence in physics-informed deep learning networks.

Findings

Physics-based regularization improves stress equilibrium accuracy.

Independent hyper-parameter tuning enhances model performance.

Different loss functions and datasets require tailored hyper-parameters.

Abstract

Incorporating scientific knowledge into deep learning (DL) models for materials-based simulations can constrain the network's predictions to be within the boundaries of the material system. Altering loss functions or adding physics-based regularization (PBR) terms to reflect material properties informs a network about the physical constraints the simulation should obey. The training and tuning process of a DL network greatly affects the quality of the model, but how this process differs when using physics-based loss functions or regularization terms is not commonly discussed. In this manuscript, several PBR methods are implemented to enforce stress equilibrium on a network predicting the stress fields of a high elastic contrast composite. Models with PBR enforced the equilibrium constraint more accurately than a model without PBR, and the stress equilibrium converged more quickly. More importantly, it was observed that independently fine-tuning each implementation resulted in more accurate models. More specifically, each loss formulation and dataset required different learning rates and loss weights for the best performance. This result has important implications on assessing the relative effectiveness of different DL models and highlights important considerations when making a comparison between DL methods.