# Probabilistic Line Searches for Stochastic Optimization

## TL;DR

This paper introduces a probabilistic line search method for stochastic optimization that uses Bayesian principles to adaptively determine step sizes, removing the need for manual learning rate tuning.

## Contribution

It develops a novel probabilistic line search algorithm combining Gaussian process surrogates with Wolfe condition beliefs, tailored for stochastic gradient descent.

## Key findings

- Effectively removes the need for manual learning rate tuning.
- Maintains low computational cost and no user parameters.
- Improves stability and efficiency in stochastic optimization.

## Abstract

In deterministic optimization, line searches are a standard tool ensuring stability and efficiency. Where only stochastic gradients are available, no direct equivalent has so far been formulated, because uncertain gradients do not allow for a strict sequence of decisions collapsing the search space. We construct a probabilistic line search by combining the structure of existing deterministic methods with notions from Bayesian optimization. Our method retains a Gaussian process surrogate of the univariate optimization objective, and uses a probabilistic belief over the Wolfe conditions to monitor the descent. The algorithm has very low computational cost, and no user-controlled parameters. Experiments show that it effectively removes the need to define a learning rate for stochastic gradient descent.