Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition

TL;DR

This paper proves that stochastic gradient descent converges quickly under a strong growth condition, achieving an $O(1/k)$ rate generally and linear convergence for strongly convex functions, expanding understanding of its efficiency.

Contribution

It demonstrates that under a specific growth condition, stochastic gradient descent attains fast convergence rates, including linear convergence for strongly convex functions.

Findings

SGD has an $O(1/k)$ convergence rate under the growth condition.

Linear convergence is achieved if the function is strongly convex.

The results extend the theoretical understanding of SGD efficiency.

Abstract

We consider optimizing a function smooth convex function $f$ that is the average of a set of differentiable functions $f_i$, under the assumption considered by Solodov [1998] and Tseng [1998] that the norm of each gradient $f_i'$ is bounded by a linear function of the norm of the average gradient $f'$. We show that under these assumptions the basic stochastic gradient method with a sufficiently-small constant step-size has an $O(1/k)$ convergence rate, and has a linear convergence rate if $g$ is strongly-convex.