Power Gradient Descent

TL;DR

This paper introduces a novel 'power gradient' modification to existing gradient descent algorithms, aiming to improve convergence speed in flat regions and stability in steep directions, with empirical validation on popular methods.

Contribution

It proposes a new power gradient approach that enhances existing gradient descent methods, including Nesterov, AMSGrad, and ADAM, demonstrating improved performance.

Findings

Significantly better performance of modified methods on benchmark tasks.

Effective integration of power gradients into ADAM for improved stability.

Empirical validation across multiple modern gradient descent algorithms.

Abstract

The development of machine learning is promoting the search for fast and stable minimization algorithms. To this end, we suggest a change in the current gradient descent methods that should speed up the motion in flat regions and slow it down in steep directions of the function to minimize. It is based on a "power gradient", in which each component of the gradient is replaced by its versus-preserving $H$-th power, with $0<H<1$. We test three modern gradient descent methods fed by such variant and by standard gradients, finding the new version to achieve significantly better performances for the Nesterov accelerated gradient and AMSGrad. We also propose an effective new take on the ADAM algorithm, which includes power gradients with varying $H$.