AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent

TL;DR

AutoSGD is a novel method that automatically adjusts the learning rate during stochastic gradient descent, reducing the need for manual tuning and improving performance across various tasks.

Contribution

We propose AutoSGD, an automatic learning rate adjustment algorithm with theoretical convergence guarantees for SGD and standard gradient descent.

Findings

AutoSGD achieves competitive or superior performance on multiple optimization problems.

Theoretical convergence of AutoSGD is established for stochastic and deterministic cases.

Empirical results demonstrate reduced tuning effort and improved efficiency.

Abstract

The learning rate is an important tuning parameter for stochastic gradient descent (SGD) and can greatly influence its performance. However, appropriate selection of a learning rate schedule across all iterations typically requires a non-trivial amount of user tuning effort. To address this, we introduce AutoSGD: an SGD method that automatically determines whether to increase or decrease the learning rate at a given iteration and then takes appropriate action. We introduce theory supporting the convergence of AutoSGD, along with its deterministic counterpart for standard gradient descent. Empirical results suggest strong performance of the method on a variety of traditional optimization problems and machine learning tasks.