Probabilistic Line Searches for Stochastic Optimization

TL;DR

This paper introduces a probabilistic line search method for stochastic optimization that uses Bayesian techniques to adaptively determine step sizes, removing the need for manual learning rate tuning.

Contribution

It develops a novel probabilistic line search algorithm combining Gaussian process surrogates with Wolfe condition beliefs, applicable to stochastic gradient descent.

Findings

Effectively removes the need for manual learning rate tuning.

Maintains low computational cost and no user parameters.

Improves stability and efficiency in stochastic optimization.

Abstract

In deterministic optimization, line searches are a standard tool ensuring stability and efficiency. Where only stochastic gradients are available, no direct equivalent has so far been formulated, because uncertain gradients do not allow for a strict sequence of decisions collapsing the search space. We construct a probabilistic line search by combining the structure of existing deterministic methods with notions from Bayesian optimization. Our method retains a Gaussian process surrogate of the univariate optimization objective, and uses a probabilistic belief over the Wolfe conditions to monitor the descent. The algorithm has very low computational cost, and no user-controlled parameters. Experiments show that it effectively removes the need to define a learning rate for stochastic gradient descent.