Convergence results for gradient flow and gradient descent systems in the artificial neural network training

TL;DR

This paper develops a rigorous mathematical convergence theory for gradient flow and gradient descent methods used in training artificial neural networks, enhancing theoretical understanding of these optimization processes.

Contribution

It provides new convergence results for continuous-time and discrete gradient-based optimization methods in neural network training.

Findings

Established convergence conditions for gradient flow equations

Proved convergence of gradient descent under certain assumptions

Enhanced mathematical understanding of neural network training dynamics

Abstract

The field of artificial neural network (ANN) training has garnered significant attention in recent years, with researchers exploring various mathematical techniques for optimizing the training process. In particular, this paper focuses on advancing the current understanding of gradient flow and gradient descent optimization methods. Our aim is to establish a solid mathematical convergence theory for continuous-time gradient flow equations and gradient descent processes based on mathematical anaylsis tools.