Adaptive Stochastic Gradient Langevin Dynamics: Taming Convergence and Saddle Point Escape Time

TL;DR

This paper introduces adaptive stochastic gradient Langevin dynamics algorithms that efficiently escape saddle points and converge to local minima in non-convex optimization, with iteration bounds nearly independent of problem dimension.

Contribution

It proposes a new adaptive Langevin dynamics framework and two specialized algorithms with improved convergence and saddle point escape guarantees.

Findings

Escape saddle points in O(log d) iterations

Converge to local minima in O(log d / ε^4) iterations

Outperforms existing first-order methods in convergence speed

Abstract

In this paper, we propose a new adaptive stochastic gradient Langevin dynamics (ASGLD) algorithmic framework and its two specialized versions, namely adaptive stochastic gradient (ASG) and adaptive gradient Langevin dynamics(AGLD), for non-convex optimization problems. All proposed algorithms can escape from saddle points with at most $O(\log d)$ iterations, which is nearly dimension-free. Further, we show that ASGLD and ASG converge to a local minimum with at most $O(\log d/\epsilon^4)$ iterations. Also, ASGLD with full gradients or ASGLD with a slowly linearly increasing batch size converge to a local minimum with iterations bounded by $O(\log d/\epsilon^2)$, which outperforms existing first-order methods.