On the Hyperparameters in Stochastic Gradient Descent with Momentum

TL;DR

This paper provides a theoretical analysis of stochastic gradient descent with momentum, highlighting the combined influence of learning rate and momentum coefficient on convergence, and explaining why momentum enhances convergence speed and robustness.

Contribution

It introduces a hyperparameters-dependent SDE framework to analyze SGD with momentum, deriving explicit convergence rates and offering new insights into the role of momentum in optimization.

Findings

The convergence rate depends on both learning rate and momentum coefficient.

Momentum improves convergence speed and robustness over standard SGD.

Nesterov momentum behaves similarly to standard momentum under noise.

Abstract

Following the same routine as [SSJ20], we continue to present the theoretical analysis for stochastic gradient descent with momentum (SGD with momentum) in this paper. Differently, for SGD with momentum, we demonstrate it is the two hyperparameters together, the learning rate and the momentum coefficient, that play the significant role for the linear rate of convergence in non-convex optimization. Our analysis is based on the use of a hyperparameters-dependent stochastic differential equation (hp-dependent SDE) that serves as a continuous surrogate for SGD with momentum. Similarly, we establish the linear convergence for the continuous-time formulation of SGD with momentum and obtain an explicit expression for the optimal linear rate by analyzing the spectrum of the Kramers-Fokker-Planck operator. By comparison, we demonstrate how the optimal linear rate of convergence and the final gap for SGD only about the learning rate varies with the momentum coefficient increasing from zero to one when the momentum is introduced. Then, we propose a mathematical interpretation why the SGD with momentum converges faster and more robust about the learning rate than the standard SGD in practice. Finally, we show the Nesterov momentum under the existence of noise has no essential difference with the standard momentum.