Neograd: Near-Ideal Gradient Descent

TL;DR

Neograd introduces a novel gradient descent method that adaptively adjusts the learning rate and, when combined with momentum, significantly outperforms Adam, achieving much lower cost function values.

Contribution

The paper presents Neograd, a new gradient descent variant that reduces plateaus and optimally adjusts learning rates, especially effective when hybridized with momentum.

Findings

Neograd outperforms Adam on several test problems.

Neograd can reach cost function values smaller by a factor of 10^8.

The hybrid NeogradM variant is particularly effective.

Abstract

The purpose of this paper is to improve upon existing variants of gradient descent by solving two problems: (1) removing (or reducing) the plateau that occurs while minimizing the cost function, (2) continually adjusting the learning rate to an "ideal" value. The approach taken is to approximately solve for the learning rate as a function of a trust metric. When this technique is hybridized with momentum, it creates an especially effective gradient descent variant, called NeogradM. It is shown to outperform Adam on several test problems, and can easily reach cost function values that are smaller by a factor of $10^8$, for example.