Stochastic Gradient Langevin Dynamics with Variance Reduction

TL;DR

This paper enhances stochastic gradient Langevin dynamics (SGLD) with variance reduction techniques, improving convergence to local minima and providing ergodicity insights for better global optimization in nonconvex problems.

Contribution

It introduces variance reduction to SGLD, proving improved convergence and ergodicity properties for nonconvex optimization.

Findings

Enhanced convergence to local minimizers

Proven ergodicity of the SGLD scheme

Insights into global minimizer potential

Abstract

Stochastic gradient Langevin dynamics (SGLD) has gained the attention of optimization researchers due to its global optimization properties. This paper proves an improved convergence property to local minimizers of nonconvex objective functions using SGLD accelerated by variance reductions. Moreover, we prove an ergodicity property of the SGLD scheme, which gives insights on its potential to find global minimizers of nonconvex objectives.