On the Compatibility between Neural Networks and Partial Differential Equations for Physics-informed Learning

TL;DR

This paper analyzes the limitations of ReLU-based neural networks in physics-informed learning and proposes a new architecture with $C^n$ activation functions that strictly satisfy linear PDEs without a loss function.

Contribution

It proves the incompatibility of ReLU networks with higher-order PDEs and introduces a novel 'out-layer-hyperplane' method for enforcing PDE constraints directly.

Findings

ReLU networks lead to a vanished Hessian, incompatible with second- or higher-order PDEs.

A new architecture with $C^n$ activations can satisfy linear PDEs exactly.

The out-layer-hyperplane enforces PDE constraints without a loss function.

Abstract

We shed light on a pitfall and an opportunity in physics-informed neural networks (PINNs). We prove that a multilayer perceptron (MLP) only with ReLU (Rectified Linear Unit) or ReLU-like Lipschitz activation functions will always lead to a vanished Hessian. Such a network-imposed constraint contradicts any second- or higher-order partial differential equations (PDEs). Therefore, a ReLU-based MLP cannot form a permissible function space for the approximation of their solutions. Inspired by this pitfall, we prove that a linear PDE up to the $n$-th order can be strictly satisfied by an MLP with $C^n$ activation functions when the weights of its output layer lie on a certain hyperplane, as called the out-layer-hyperplane. An MLP equipped with the out-layer-hyperplane becomes "physics-enforced", no longer requiring a loss function for the PDE itself (but only those for the initial and boundary conditions). Such a hyperplane exists not only for MLPs but for any network architecture tailed by a fully-connected hidden layer. To our knowledge, this should be the first PINN architecture that enforces point-wise correctness of PDEs. We show a closed-form expression of the out-layer-hyperplane for second-order linear PDEs, which can be generalised to higher-order nonlinear PDEs.