On the overfly algorithm in deep learning of neural networks

TL;DR

This paper explores the supervised training of neural networks using a dynamical systems perspective, introducing the overfly algorithm to address local minima issues in backpropagation.

Contribution

It presents the overfly algorithm, a novel method inspired by differential equations, to improve neural network training by mitigating local minima problems.

Findings

Overfly algorithm effectively reduces local minima during training.

Dynamical systems approach offers new insights into neural network optimization.

Theoretical analysis links differential equations to neural network training dynamics.

Abstract

In this paper we investigate the supervised backpropagation training of multilayer neural networks from a dynamical systems point of view. We discuss some links with the qualitative theory of differential equations and introduce the overfly algorithm to tackle the local minima problem. Our approach is based on the existence of first integrals of the generalised gradient system with build-in dissipation.