Polyak's Heavy Ball Method Achieves Accelerated Local Rate of Convergence under Polyak-Lojasiewicz Inequality

TL;DR

This paper analyzes Polyak's heavy ball method under the Polyak-Lojasiewicz inequality, showing it achieves accelerated local convergence rates in non-convex settings with a novel geometric approach.

Contribution

It provides the first local convergence analysis of the heavy ball method under the Polyak-Lojasiewicz inequality using a differential geometric perspective.

Findings

Heavy ball method exhibits local acceleration under PL inequality.

Local convergence occurs near minima even with aggressive hyperparameters.

New geometric approach replaces Lyapunov-based analysis.

Abstract

In this work, we analyze the convergence of Polyak's heavy ball method in both continuous and discrete time for non-convex $C^4$-objective functions satisfying the Polyak-Lojasiewicz inequality. Under this weak assumption, we recover the asymptotic convergence rates originally derived by Polyak in [Polyak, U.S.S.R. Comput. Math. and Math. Phys., 1964] for strongly convex objectives. Our results demonstrate that the heavy ball method exhibits asymptotic local acceleration on this class of functions. In particular, in the discrete time setting, we prove local convergence of the iterates to a minimum once the method enters a sufficiently small neighborhood of the set of minima, for a broad range of hyperparameters, including aggressive choices for the momentum parameter and the step-size for which global convergence is known to fail. Instead of the usually employed Lyapunov-type arguments, our approach leverages a new differential geometric perspective of the Polyak-Lojasiewicz inequality proposed in [Rebjock and Boumal, Math. Program., 2025].