Faster Biological Gradient Descent Learning

TL;DR

This paper introduces DLR, a biologically inspired local gradient descent algorithm that adjusts learning rates based on connection weights, significantly speeding up training especially in small neural networks.

Contribution

The paper presents a simple, local, and biologically plausible gradient descent method called DLR that improves training speed without requiring past update information.

Findings

DLR accelerates training in small neural networks.

DLR is biologically plausible and easy to implement.

It outperforms traditional gradient descent in convergence speed.

Abstract

Back-propagation is a popular machine learning algorithm that uses gradient descent in training neural networks for supervised learning, but can be very slow. A number of algorithms have been developed to speed up convergence and improve robustness of the learning. However, they are complicated to implement biologically as they require information from previous updates. Inspired by synaptic competition in biology, we have come up with a simple and local gradient descent optimization algorithm that can reduce training time, with no demand on past details. Our algorithm, named dynamic learning rate (DLR), works similarly to the traditional gradient descent used in back-propagation, except that instead of having a uniform learning rate across all synapses, the learning rate depends on the current neuronal connection weights. Our algorithm is found to speed up learning, particularly for small networks.