Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search

TL;DR

This paper compares two recent learning rate optimization methods for stochastic gradient descent, D-Adaptation and probabilistic line search, highlighting their shared goals and potential for integration.

Contribution

It provides an intuitive overview of both methods, analyzes their similarities, and explores the possibility of merging these approaches for improved learning rate adaptation.

Findings

Both methods aim to automate learning rate selection.

They utilize distance metrics and Gaussian process estimates.

Potential for combining the methods to enhance optimization.

Abstract

This paper explores two recent methods for learning rate optimisation in stochastic gradient descent: D-Adaptation (arXiv:2301.07733) and probabilistic line search (arXiv:1502.02846). These approaches aim to alleviate the burden of selecting an initial learning rate by incorporating distance metrics and Gaussian process posterior estimates, respectively. In this report, I provide an intuitive overview of both methods, discuss their shared design goals, and devise scope for merging the two algorithms.