One-dimensional System Arising in Stochastic Gradient Descent

TL;DR

This paper analyzes stochastic differential equations modeling one-dimensional stochastic gradient descent, identifying conditions under which the process converges to zero or escapes saddle points, depending on the decay rate parameter.

Contribution

It establishes a threshold for the decay parameter in SDEs related to SGD, determining convergence or escape from saddle points based on the function's behavior.

Findings

Existence of a threshold  for  determining convergence to zero.

Extension of results to discrete processes with bounded martingale differences.

Ability to escape saddle points for functions with polynomial order second derivatives.

Abstract

We consider SDEs of the form $dX_t = |f(X_t)|/t^{\gamma} dt+1/t^{\gamma} dB_t$, where $f(x)$ behaves comparably to $|x|^k$ in a neighborhood of the origin, for $k\in [1,\infty)$. We show that there exists a threshold value $:=\tilde{\gamma}$ for $\gamma$, depending on $k$, such that when $\gamma \in (1/2, \tilde{\gamma})$ then $\mathbb{P}(X_n\rightarrow 0) = 0$, and for the rest of the permissible values $\mathbb{P}(X_n\rightarrow 0)>0$. The previous results extend for discrete processes that satisfy $X_{n+1}-X_n = f(X_n)/n^\gamma +Y_n/n^\gamma$. Here, $Y_{n+1}$ are martingale differences that are a.s. bounded. This result shows that for a function $F$, whose second derivative at degenerate saddle points is of polynomial order, it is always possible to escape saddle points via the iteration $X_{n+1}-X_n =F'(X_n)/n^\gamma +Y_n/n^\gamma$ for a suitable choice of $\gamma$.