A Parameter-Free Stochastic LineseArch Method (SLAM) for Minimizing Expectation Residuals

TL;DR

This paper introduces a parameter-free stochastic line search method that adaptively determines step sizes without knowing problem parameters, achieving optimal convergence rates for smooth and nonsmooth convex optimization.

Contribution

It proposes a novel Armijo-enabled stochastic linesearch framework that removes the need for prior knowledge of Lipschitz constants, with proven convergence guarantees.

Findings

Expected stationarity residual decreases at rate O(1/√K)

Iteration complexity is O(ε^{-2}) for ε-stationary points

Sample complexity is O(ε^{-4}) for ε-stationary points

Abstract

Most existing rate and complexity guarantees for stochastic gradient methods in $L$-smooth settings mandates that such sequences be non-adaptive, non-increasing, and upper bounded by $\tfrac{a}{L}$ for $a > 0$. This requires knowledge of $L$ and may preclude larger steps. Motivated by these shortcomings, we present an Armijo-enabled stochastic linesearch framework with standard stochastic zeroth- and first-order oracles. The resulting steplength sequence is non-monotone and requires neither knowledge of $L$ nor any other problem parameters. We then prove that the expected stationarity residual diminishes at a rate of $\mathcal{O}(1/\sqrt{K})$, where $K$ denotes the iteration budget. Furthermore, the resulting iteration and sample complexities for computing an $\epsilon$-stationary point are $\mathcal{O}(\epsilon^{-2})$ and $\mathcal{O}\left(\epsilon^{-4}\right)$. The proposed method allows for a simple nonsmooth convex component in the objective, addressed through proximal gradient updates. Analogous guarantees are provided in the Polyak-Lojasiewicz (PL) setting and convex regimes. Preliminary numerical experiments are seen to be promising.