Accelerated Almost-Sure Convergence Rates for Nonconvex Stochastic Gradient Descent using Stochastic Learning Rates

TL;DR

This paper demonstrates that incorporating stochastic learning rates into Stochastic Gradient Descent accelerates almost-sure convergence in nonconvex optimization, supported by theoretical analysis and empirical validation.

Contribution

It introduces a stochastic learning rate scheme for SGD that achieves faster almost-sure convergence rates in nonconvex problems, advancing optimization theory.

Findings

Accelerated convergence rates with stochastic learning rates

Theoretical proof of improved almost-sure convergence

Empirical validation confirming theoretical results

Abstract

Large-scale optimization problems require algorithms both effective and efficient. One such popular and proven algorithm is Stochastic Gradient Descent which uses first-order gradient information to solve these problems. This paper studies almost-sure convergence rates of the Stochastic Gradient Descent method when instead of deterministic, its learning rate becomes stochastic. In particular, its learning rate is equipped with a multiplicative stochasticity, producing a stochastic learning rate scheme. Theoretical results show accelerated almost-sure convergence rates of Stochastic Gradient Descent in a nonconvex setting when using an appropriate stochastic learning rate, compared to a deterministic-learning-rate scheme. The theoretical results are verified empirically.