Averaged Heavy-Ball Method

TL;DR

This paper introduces an averaged version of the Heavy-Ball method (AHB) that reduces deviation and improves convergence properties, addressing the peak effect issue in the original method, with theoretical analysis and numerical validation.

Contribution

The paper proposes AHB, an averaged Heavy-Ball method, providing theoretical convergence guarantees and demonstrating practical advantages over the standard HB method.

Findings

AHB has smaller maximal deviation than HB on quadratic problems.

AHB achieves non-accelerated convergence rates for convex functions.

Numerical experiments show AHB's improved performance in practice.

Abstract

Heavy-Ball method (HB) is known for its simplicity in implementation and practical efficiency. However, as with other momentum methods, it has non-monotone behavior, and for optimal parameters, the method suffers from the so-called peak effect. To address this issue, in this paper, we consider an averaged version of Heavy-Ball method (AHB). We show that for quadratic problems AHB has a smaller maximal deviation from the solution than HB. Moreover, for general convex and strongly convex functions, we prove non-accelerated rates of global convergence of AHB and its weighted version. We conduct several numerical experiments on minimizing quadratic and non-quadratic functions to demonstrate the advantages of using averaging for HB.