Enhancing approximation abilities of neural networks by training derivatives

TL;DR

This paper introduces a method to improve neural network approximation accuracy by incorporating derivatives into the training process, significantly enhancing precision especially for differential equations.

Contribution

It presents a novel training approach that uses target derivatives in the cost function, enabling neural networks to achieve much higher accuracy with a GPU-efficient gradient calculation.

Findings

Achieves 140-1000 times more accurate approximation with equal operations.

Enables solving differential equations with 13 times better accuracy.

Allows larger grid steps in differential equation solutions.

Abstract

A method to increase the precision of feedforward networks is proposed. It requires a prior knowledge of a target function derivatives of several orders and uses this information in gradient based training. Forward pass calculates not only the values of the output layer of a network but also their derivatives. The deviations of those derivatives from the target ones are used in an extended cost function and then backward pass calculates the gradient of the extended cost with respect to weights, which can then be used by any weights update algorithm. Despite a substantial increase in arithmetic operations per pattern (if compared to the conventional training), the extended cost allows to obtain 140--1000 times more accurate approximation for simple cases if the total number of operations is equal. This precision also happens to be out of reach for the regular cost function. The method fits well into the procedure of solving differential equations with neural networks. Unlike training a network to match some target mapping, which requires an explicit use of the target derivatives in the extended cost function, the cost function for solving a differential equation is based on the deviation of the equation's residual from zero and thus can be extended by differentiating the equation itself, which does not require any prior knowledge. Solving an equation with such a cost resulted in 13 times more accurate result and could be done with 3 times larger grid step. GPU-efficient algorithm for calculating the gradient of the extended cost function is proposed.