A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks

TL;DR

This paper introduces n-TangentProp, a quasilinear algorithm for efficiently computing high-order derivatives of neural networks, significantly improving speed over traditional exponential-time autodifferentiation methods.

Contribution

The paper presents n-TangentProp, extending TangentProp to compute derivatives of arbitrary order in quasilinear time for feed-forward neural networks.

Findings

n-TangentProp computes derivatives faster than previous methods.

The algorithm scales well with network size and derivative order.

It enables faster training of physics-informed neural networks.

Abstract

The use of neural networks for solving differential equations is practically difficult due to the exponentially increasing runtime of autodifferentiation when computing high-order derivatives. We propose $n$-TangentProp, the natural extension of the TangentProp formalism \cite{simard1991tangent} to arbitrarily many derivatives. $n$-TangentProp computes the exact derivative $d^n/dx^n f(x)$ in quasilinear, instead of exponential time, for a densely connected, feed-forward neural network $f$ with a smooth, parameter-free activation function. We validate our algorithm empirically across a range of depths, widths, and number of derivatives. We demonstrate that our method is particularly beneficial in the context of physics-informed neural networks where \ntp allows for significantly faster training times than previous methods and has favorable scaling with respect to both model size and loss-function complexity as measured by the number of required derivatives. The code for this paper can be found at https://github.com/kyrochi/n\_tangentprop.