Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descent

TL;DR

This paper introduces a novel analytical framework using modified stochastic differential equations to better understand the qualitative behavior and stability of stochastic optimization algorithms, specifically stochastic coordinate descent.

Contribution

It applies backward error analysis to stochastic optimization, providing new insights into the stability and dynamics of these methods through modified equations.

Findings

Modified stochastic differential equations closely approximate the dynamics of stochastic optimization methods.

Analysis reveals mean-square stability conditions for stochastic coordinate descent.

Provides qualitative insights into the behavior of stochastic algorithms.

Abstract

Stochastic optimization methods have been hugely successful in making large-scale optimization problems feasible when computing the full gradient is computationally prohibitive. Using the theory of modified equations for numerical integrators, we propose a class of stochastic differential equations that approximate the dynamics of general stochastic optimization methods more closely than the original gradient flow. Analyzing a modified stochastic differential equation can reveal qualitative insights about the associated optimization method. Here, we study mean-square stability of the modified equation in the case of stochastic coordinate descent.