Explainable Learning Rate Regimes for Stochastic Optimization

TL;DR

This paper introduces an explainable, automatic learning rate regime for stochastic gradient descent that adapts based on the intrinsic variation of stochastic gradients, improving efficiency and robustness without extra hyper-parameter tuning.

Contribution

It develops a novel, explainable learning rate adjustment method leveraging stochastic second-order algorithms that requires no manual hyper-parameter tuning.

Findings

Demonstrates efficiency across multiple stochastic algorithms

Shows robustness in various machine learning tasks

Scales effectively with different models and datasets

Abstract

Modern machine learning is trained by stochastic gradient descent (SGD), whose performance critically depends on how the learning rate (LR) is adjusted and decreased over time. Yet existing LR regimes may be intricate, or need to tune one or more additional hyper-parameters manually whose bottlenecks include huge computational expenditure, time and power in practice. This work, in a natural and direct manner, clarifies how LR should be updated automatically only according to the intrinsic variation of stochastic gradients. An explainable LR regime by leveraging stochastic second-order algorithms is developed, behaving a similar pattern to heuristic algorithms but implemented simply without any parameter tuning requirement, where it is of an automatic procedure that LR should increase (decrease) as the norm of stochastic gradients decreases (increases). The resulting LR regime shows its efficiency, robustness, and scalability in different classical stochastic algorithms, containing SGD, SGDM, and SIGNSGD, on machine learning tasks.