Error estimates between SGD with momentum and underdamped Langevin diffusion

TL;DR

This paper provides a quantitative analysis of the error between stochastic gradient descent with momentum and underdamped Langevin diffusion, highlighting their close relationship in probabilistic metrics.

Contribution

It establishes a novel error estimate between SGD with momentum and underdamped Langevin diffusion in Wasserstein and total variation distances.

Findings

Error bounds in 1-Wasserstein distance

Error bounds in total variation distance

Quantitative comparison of SGD with momentum and Langevin diffusion

Abstract

Stochastic gradient descent with momentum is a popular variant of stochastic gradient descent, which has recently been reported to have a close relationship with the underdamped Langevin diffusion. In this paper, we establish a quantitative error estimate between them in the 1-Wasserstein and total variation distances.