Online Learning Rate Adaptation with Hypergradient Descent

TL;DR

This paper presents a simple, practical method for automatically adapting learning rates during optimization by using hypergradients, improving convergence without manual tuning across various algorithms.

Contribution

It introduces a hypergradient-based approach for dynamic learning rate adjustment that is easy to implement and effective in practice, applicable to multiple optimizers.

Findings

Reduces manual learning rate tuning

Improves convergence speed across optimizers

Requires minimal additional computation

Abstract

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov momentum, and Adam, showing that it significantly reduces the need for the manual tuning of the initial learning rate for these commonly used algorithms. Our method works by dynamically updating the learning rate during optimization using the gradient with respect to the learning rate of the update rule itself. Computing this "hypergradient" needs little additional computation, requires only one extra copy of the original gradient to be stored in memory, and relies upon nothing more than what is provided by reverse-mode automatic differentiation.