On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems

TL;DR

This paper provides a comprehensive analysis of stochastic gradient descent (SGD) in non-convex problems, establishing almost sure convergence, avoidance of saddle points, and convergence rates, with practical insights for step-size tuning.

Contribution

It proves almost sure convergence of SGD in non-convex settings, shows SGD avoids saddle points with probability 1, and derives convergence rates for Hurwicz minimizers under various step-size schedules.

Findings

SGD iterates remain bounded and converge with probability 1.

SGD avoids strict saddle points with probability 1.

Convergence rate to Hurwicz minimizers is O(1/n^p) with step-size Θ(1/n^p).

Abstract

This paper analyzes the trajectories of stochastic gradient descent (SGD) to help understand the algorithm's convergence properties in non-convex problems. We first show that the sequence of iterates generated by SGD remains bounded and converges with probability $1$ under a very broad range of step-size schedules. Subsequently, going beyond existing positive probability guarantees, we show that SGD avoids strict saddle points/manifolds with probability $1$ for the entire spectrum of step-size policies considered. Finally, we prove that the algorithm's rate of convergence to Hurwicz minimizers is $\mathcal{O}(1/n^{p})$ if the method is employed with a $\Theta(1/n^p)$ step-size schedule. This provides an important guideline for tuning the algorithm's step-size as it suggests that a cool-down phase with a vanishing step-size could lead to faster convergence; we demonstrate this heuristic using ResNet architectures on CIFAR.