On the Convergence Analysis of Aggregated Heavy-Ball Method

TL;DR

This paper provides the first convergence analysis of the Aggregated Heavy-Ball method for smooth, non-convex, convex, and strongly convex functions, demonstrating its theoretical robustness and practical efficiency.

Contribution

It extends the analysis of AggHB beyond quadratic and bounded gradient assumptions to general smooth objectives, matching known complexity results.

Findings

AggHB is more stable than classical Heavy-Ball with large momentum.

Theoretical convergence rates for AggHB are established for various convexity settings.

Numerical experiments confirm AggHB's efficiency on non-convex and convex problems.

Abstract

Momentum first-order optimization methods are the workhorses in various optimization tasks, e.g., in the training of deep neural networks. Recently, Lucas et al. (2019) proposed a method called Aggregated Heavy-Ball (AggHB) that uses multiple momentum vectors corresponding to different momentum parameters and averages these vectors to compute the update direction at each iteration. Lucas et al. (2019) show that AggHB is more stable than the classical Heavy-Ball method even with large momentum parameters and performs well in practice. However, the method was analyzed only for quadratic objectives and for online optimization tasks under uniformly bounded gradients assumption, which is not satisfied for many practically important problems. In this work, we address this issue and propose the first analysis of AggHB for smooth objective functions in non-convex, convex, and strongly convex cases without additional restrictive assumptions. Our complexity results match the best-known ones for the Heavy-Ball method. We also illustrate the efficiency of AggHB numerically on several non-convex and convex problems.