Backpropagation generalized for output derivatives

TL;DR

This paper extends backpropagation to compute derivatives of neural network outputs with respect to inputs, enabling neural networks to solve differential equations exactly without approximation, suitable for parallel implementation.

Contribution

It introduces a generalized backpropagation method for training neural networks to compute output derivatives, applicable to solving PDEs with exact differential operators.

Findings

Enables neural networks to solve differential equations directly.

Provides a matrix-vector form for efficient parallel computation.

Extends to N-dimensional derivatives for complex PDEs.

Abstract

Backpropagation algorithm is the cornerstone for neural network analysis. Paper extends it for training any derivatives of neural network's output with respect to its input. By the dint of it feedforward networks can be used to solve or verify solutions of partial or simple, linear or nonlinear differential equations. This method vastly differs from traditional ones like finite differences on a mesh. It contains no approximations, but rather an exact form of differential operators. Algorithm is built to train a feed forward network with any number of hidden layers and any kind of sufficiently smooth activation functions. It's presented in a form of matrix-vector products so highly parallel implementation is readily possible. First part derives the method for 2D case with first and second order derivatives, second part extends it to N-dimensional case with any derivatives. All necessary expressions for using this method to solve most applied PDE can be found in Appendix D.