Are Two Hidden Layers Still Enough for the Physics-Informed Neural Networks?

TL;DR

This paper explores the capabilities of neural networks with one or two hidden layers for solving physical problems described by ODEs and PDEs, introducing new initialization, training, and generalization methods that achieve competitive or state-of-the-art accuracy.

Contribution

It proposes novel deterministic and data-driven initialization methods, training techniques, and a gradient-free fitting approach for shallow physics-informed neural networks, extending them to 2D problems.

Findings

Methods achieve competitive accuracy on ODEs and PDEs

Neural networks with one or two hidden layers can be highly effective

Proposed approaches outperform some existing techniques in certain cases

Abstract

The article discusses the development of various methods and techniques for initializing and training neural networks with a single hidden layer, as well as training a separable physics-informed neural network consisting of neural networks with a single hidden layer to solve physical problems described by ordinary differential equations (ODEs) and partial differential equations (PDEs). A method for strictly deterministic initialization of a neural network with one hidden layer for solving physical problems described by an ODE is proposed. Modifications to existing methods for weighting the loss function are given, as well as new methods developed for training strictly deterministic-initialized neural networks to solve ODEs (detaching, additional weighting based on the second derivative, predicted solution-based weighting, relative residuals). An algorithm for physics-informed data-driven initialization of a neural network with one hidden layer is proposed. A neural network with pronounced generalizing properties is presented, whose generalizing abilities of which can be precisely controlled by adjusting network parameters. A metric for measuring the generalization of such neural network has been introduced. A gradient-free neuron-by-neuron fitting method has been developed for adjusting the parameters of a single-hidden-layer neural network, which does not require the use of an optimizer or solver for its implementation. The proposed methods have been extended to 2D problems using the separable physics-informed neural networks approach. Numerous experiments have been carried out to develop the above methods and approaches. Experiments on physical problems, such as solving various ODEs and PDEs, have demonstrated that these methods for initializing and training neural networks with one or two hidden layers (SPINN) achieve competitive accuracy and, in some cases, state-of-the-art results.