When and Why Momentum Accelerates SGD:An Empirical Study

TL;DR

This paper empirically investigates when and why momentum accelerates SGD, revealing that momentum's benefits are linked to effective learning rates and the prevention of abrupt sharpening, especially with larger batch sizes.

Contribution

It introduces a comparison framework based on effective learning rates and uncovers the role of abrupt sharpening in momentum acceleration during SGD training.

Findings

SGDM and SGD perform similarly at low effective learning rates.

SGDM outperforms SGD when effective learning rates exceed a threshold.

Momentum helps prevent abrupt sharpening, improving convergence especially with larger batch sizes.

Abstract

Momentum has become a crucial component in deep learning optimizers, necessitating a comprehensive understanding of when and why it accelerates stochastic gradient descent (SGD). To address the question of ''when'', we establish a meaningful comparison framework that examines the performance of SGD with Momentum (SGDM) under the \emph{effective learning rates} $\eta_{ef}$, a notion unifying the influence of momentum coefficient $\mu$ and batch size $b$ over learning rate $\eta$. In the comparison of SGDM and SGD with the same effective learning rate and the same batch size, we observe a consistent pattern: when $\eta_{ef}$ is small, SGDM and SGD experience almost the same empirical training losses; when $\eta_{ef}$ surpasses a certain threshold, SGDM begins to perform better than SGD. Furthermore, we observe that the advantage of SGDM over SGD becomes more pronounced with a larger batch size. For the question of ``why'', we find that the momentum acceleration is closely related to \emph{abrupt sharpening} which is to describe a sudden jump of the directional Hessian along the update direction. Specifically, the misalignment between SGD and SGDM happens at the same moment that SGD experiences abrupt sharpening and converges slower. Momentum improves the performance of SGDM by preventing or deferring the occurrence of abrupt sharpening. Together, this study unveils the interplay between momentum, learning rates, and batch sizes, thus improving our understanding of momentum acceleration.