A Derivation of Feedforward Neural Network Gradients Using Fr\'echet Calculus

TL;DR

This paper introduces a compact derivation of neural network gradients using Fréchet calculus, providing a unified approach applicable to various architectures including convolutional networks.

Contribution

It presents a novel, more concise derivation of neural network gradients via Fréchet calculus, extending to complex architectures.

Findings

Derivation of neural network gradients using Fréchet calculus

Development of an efficient algorithm for gradient computation

Generalization to convolutional and other advanced architectures

Abstract

We present a derivation of the gradients of feedforward neural networks using Fr\'echet calculus which is arguably more compact than the ones usually presented in the literature. We first derive the gradients for ordinary neural networks working on vectorial data and show how these derived formulas can be used to derive a simple and efficient algorithm for calculating a neural networks gradients. Subsequently we show how our analysis generalizes to more general neural network architectures including, but not limited to, convolutional networks.