Continuous-time Models for Stochastic Optimization Algorithms

TL;DR

This paper introduces continuous-time models for stochastic optimization algorithms, providing new insights into their convergence behavior and dynamics, especially for non-convex functions, using stochastic calculus and Lyapunov analysis.

Contribution

It develops novel continuous-time formulations for stochastic algorithms and applies stochastic calculus to analyze their convergence and dynamics, bridging discrete and continuous perspectives.

Findings

Derived convergence bounds for non-convex functions.

Showed that decreasing learning rate acts as landscape stretching.

Matched rates between continuous-time models and discrete algorithms.

Abstract

We propose new continuous-time formulations for first-order stochastic optimization algorithms such as mini-batch gradient descent and variance-reduced methods. We exploit these continuous-time models, together with simple Lyapunov analysis as well as tools from stochastic calculus, in order to derive convergence bounds for various types of non-convex functions. Guided by such analysis, we show that the same Lyapunov arguments hold in discrete-time, leading to matching rates. In addition, we use these models and Ito calculus to infer novel insights on the dynamics of SGD, proving that a decreasing learning rate acts as time warping or, equivalently, as landscape stretching.