Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activations

TL;DR

This paper analyzes the residual loss landscape of Physics-Informed Neural Networks (PINNs), revealing conditions under which wide networks and specific activation functions with well-behaved derivatives optimize training for solving PDEs.

Contribution

It provides a theoretical analysis of residual loss in PINNs, highlighting the importance of network width and activation function properties for effective training.

Findings

Wide neural networks can globally minimize residual loss under certain conditions.

Activation functions with well-behaved high-order derivatives are crucial for minimizing residual loss.

Periodic activations explain their promising performance in PINNs.

Abstract

The residual loss in Physics-Informed Neural Networks (PINNs) alters the simple recursive relation of layers in a feed-forward neural network by applying a differential operator, resulting in a loss landscape that is inherently different from those of common supervised problems. Therefore, relying on the existing theory leads to unjustified design choices and suboptimal performance. In this work, we analyze the residual loss by studying its characteristics at critical points to find the conditions that result in effective training of PINNs. Specifically, we first show that under certain conditions, the residual loss of PINNs can be globally minimized by a wide neural network. Furthermore, our analysis also reveals that an activation function with well-behaved high-order derivatives plays a crucial role in minimizing the residual loss. In particular, to solve a $k$-th order PDE, the $k$-th derivative of the activation function should be bijective. The established theory paves the way for designing and choosing effective activation functions for PINNs and explains why periodic activations have shown promising performance in certain cases. Finally, we verify our findings by conducting a set of experiments on several PDEs. Our code is publicly available at https://github.com/nimahsn/pinns_tf2.