A Stochastic Line Search Method with Convergence Rate Analysis

TL;DR

This paper introduces a stochastic line search method that adapts classical techniques to stochastic optimization, providing convergence guarantees and efficiency comparable to deterministic methods under probabilistic accuracy assumptions.

Contribution

It proposes a novel stochastic line search algorithm with convergence rate analysis, extending classical deterministic line search to stochastic settings with probabilistic accuracy.

Findings

Expected iterations match worst-case efficiency of first-order methods.

Achieves deterministic gradient descent rates for convex objectives.

Provides convergence guarantees under probabilistic accuracy assumptions.

Abstract

For deterministic optimization, line-search methods augment algorithms by providing stability and improved efficiency. We adapt a classical backtracking Armijo line-search to the stochastic optimization setting. While traditional line-search relies on exact computations of the gradient and values of the objective function, our method assumes that these values are available up to some dynamically adjusted accuracy which holds with some sufficiently large, but fixed, probability. We show the expected number of iterations to reach a near stationary point matches the worst-case efficiency of typical first-order methods, while for convex and strongly convex objective, it achieves rates of deterministic gradient descent in function values.