Stochastic Learning Rate Optimization in the Stochastic Approximation and Online Learning Settings

TL;DR

This paper introduces a stochastic learning rate scheme for optimization algorithms, providing theoretical convergence guarantees and empirical evidence of improved performance over deterministic learning rates.

Contribution

It proposes a novel stochastic learning rate method with convergence analysis and demonstrates its effectiveness across various algorithms.

Findings

Noticeable optimization performance gains with stochastic learning rates

Theoretical convergence results under stochastic and online settings

Empirical improvements in multiple algorithms using stochastic learning rates

Abstract

In this work, multiplicative stochasticity is applied to the learning rate of stochastic optimization algorithms, giving rise to stochastic learning-rate schemes. In-expectation theoretical convergence results of Stochastic Gradient Descent equipped with this novel stochastic learning rate scheme under the stochastic setting, as well as convergence results under the online optimization settings are provided. Empirical results consider the case of an adaptively uniformly distributed multiplicative stochasticity and include not only Stochastic Gradient Descent, but also other popular algorithms equipped with a stochastic learning rate. They demonstrate noticeable optimization performance gains, with respect to their deterministic-learning-rate versions.