Stochastic Gradient Descent Revisited

TL;DR

This paper provides a comprehensive convergence analysis of biased nonconvex stochastic gradient descent, covering various convergence types and rates under mild assumptions, enhancing theoretical understanding of SGD in machine learning.

Contribution

It offers a full scope convergence study of biased nonconvex SGD, including weak, function-value, and global convergence, with convergence rates and complexities under mild conditions.

Findings

Established weak convergence of biased nonconvex SGD

Derived convergence rates and complexities

Provided conditions under which convergence guarantees hold

Abstract

Stochastic gradient descent (SGD) has been a go-to algorithm for nonconvex stochastic optimization problems arising in machine learning. Its theory however often requires a strong framework to guarantee convergence properties. We hereby present a full scope convergence study of biased nonconvex SGD, including weak convergence, function-value convergence and global convergence, and also provide subsequent convergence rates and complexities, all under relatively mild conditions in comparison with literature.