A Stochastic Gradient Descent Theorem and the Back-Propagation Algorithm

TL;DR

This paper proves a convergence theorem for a specific stochastic gradient descent method, providing theoretical foundations for a convergent version of the back-propagation algorithm used in training neural networks.

Contribution

It introduces a new convergence theorem for stochastic gradient descent, enhancing the theoretical understanding of back-propagation algorithms.

Findings

Proves convergence of a stochastic gradient descent variant.

Provides theoretical validation for back-propagation.

Lays groundwork for more reliable neural network training methods.

Abstract

We establish a convergence theorem for a certain type of stochastic gradient descent, which leads to a convergent variant of the back-propagation algorithm