Preconditioning and Numerical Stability in Neural Network Training for Parametric PDEs

TL;DR

This paper explores how preconditioning with well-conditioned frames improves neural network training for parametric PDEs, ensuring numerical stability and enabling efficient low-precision computations.

Contribution

It introduces a novel stable representation of preconditioned operators that maintains precision in low-precision floating point formats.

Findings

Preconditioning significantly enhances training performance.

Stable operator representations enable low-precision computations without loss of accuracy.

Standard representations are insufficient for numerical stability.

Abstract

In the context of training neural network-based approximations of solutions of parameter-dependent PDEs, we investigate the effect of preconditioning via well-conditioned frame representations of operators and demonstrate a significant improvement on the performance of standard training methods. We also observe that standard representations of preconditioned matrices are insufficient for obtaining numerical stability and propose a generally applicable form of stable representations that enables computations with single- and half-precision floating point numbers without loss of precision.