Reparametrizing gradient descent

TL;DR

This paper introduces norm-adapted gradient descent, an optimization algorithm that adjusts learning rates based on a first-order estimate of gradient effects, aiming to efficiently find roots in models with sufficient capacity.

Contribution

The paper proposes a novel gradient descent variant that estimates the effect of updates to adapt learning rates, differing from statistical-based methods like Adam or Adagrad.

Findings

Strong performance in regression tasks

Capable of training classifiers

Effective in models with sufficient capacity

Abstract

In this work, we propose an optimization algorithm which we call norm-adapted gradient descent. This algorithm is similar to other gradient-based optimization algorithms like Adam or Adagrad in that it adapts the learning rate of stochastic gradient descent at each iteration. However, rather than using statistical properties of observed gradients, norm-adapted gradient descent relies on a first-order estimate of the effect of a standard gradient descent update step, much like the Newton-Raphson method in many dimensions. Our algorithm can also be compared to quasi-Newton methods, but we seek roots rather than stationary points. Seeking roots can be justified by the fact that for models with sufficient capacity measured by nonnegative loss functions, roots coincide with global optima. This work presents several experiments where we have used our algorithm; in these results, it appears norm-adapted descent is particularly strong in regression settings but is also capable of training classifiers.