Convergence Analysis of Stochastic Accelerated Gradient Methods for Generalized Smooth Optimizations

TL;DR

This paper analyzes the convergence rates of the RSAG method for stochastic optimization with generalized smooth functions, providing high-probability bounds under relaxed noise assumptions and extending results to SGD.

Contribution

It introduces convergence guarantees for RSAG with constant or adaptive step sizes under relaxed noise conditions, improving understanding of stochastic accelerated methods.

Findings

High-probability convergence rate of (\u007F(rac{\u221a{\u0131}(rac{1}{\u03b4})}{T})) for convex functions.

Improved convergence rate of (rac{\u221a{\u0131}(rac{1}{\u03b4})}{T}) when noise is small.

Applicability of analysis to SGD with various step size strategies.

Abstract

We investigate the Randomized Stochastic Accelerated Gradient (RSAG) method, utilizing either constant or adaptive step sizes, for stochastic optimization problems with generalized smooth objective functions. Under relaxed affine variance assumptions for the stochastic gradient noise, we establish high-probability convergence rates of order $\tilde{O}\left(\sqrt{\log(1/\delta)/T}\right)$ for function value gaps in the convex setting, and for the squared gradient norms in the non-convex setting. Furthermore, when the noise parameters are sufficiently small, the convergence rate improves to $\tilde{O}\left(\log(1/\delta)/T\right)$, where $T$ denotes the total number of iterations and $\delta$ is the probability margin. Our analysis is also applicable to SGD with both constant and adaptive step sizes.