Adaptive SGD with Line-Search and Polyak Stepsizes: Nonconvex Convergence and Accelerated Rates

TL;DR

This paper provides a unified convergence analysis for adaptive stochastic gradient methods with line-search and Polyak stepsizes, demonstrating improved rates for nonconvex optimization under various conditions.

Contribution

It extends convergence guarantees of AdaSLS and AdaSPS to nonconvex settings, establishing new rates under different assumptions.

Findings

Achieves $oxed{O(1/\sqrt{T})}$ convergence for general nonconvex functions.

Attains $oxed{O(1/T)}$ rates under quasar-convexity and interpolation.

Provides $oxed{O(1/T)}$ convergence under the strong growth condition.

Abstract

We extend the convergence analysis of AdaSLS and AdaSPS in [Jiang and Stich, 2024] to the nonconvex setting, presenting a unified convergence analysis of stochastic gradient descent with adaptive Armijo line-search (AdaSLS) and Polyak stepsize (AdaSPS) for nonconvex optimization. Our contributions include: (1) an $\mathcal{O}(1/\sqrt{T})$ convergence rate for general nonconvex smooth functions, (2) an $\mathcal{O}(1/T)$ rate under quasar-convexity and interpolation, and (3) an $\mathcal{O}(1/T)$ rate under the strong growth condition for general nonconvex functions.