Linearly convergent stochastic heavy ball method for minimizing generalization error

TL;DR

This paper proves the first linear convergence rate for a stochastic heavy ball method using fixed stepsize and momentum, specifically for minimizing expected loss in quadratic settings, expanding understanding of its efficiency.

Contribution

It introduces the first linear convergence analysis for the stochastic heavy ball method on expected loss minimization with fixed stepsize and momentum.

Findings

First linear convergence result established for stochastic heavy ball method.

Analysis focuses on expected loss, not finite-sum problems.

Applicable to quadratic loss, even if the overall objective isn't strongly convex.

Abstract

In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder problem. While in the analysis we constrain ourselves to quadratic loss, the overall objective is not necessarily strongly convex.