An Accelerated Lyapunov Function for Polyak's Heavy-Ball on Convex Quadratics

TL;DR

This paper introduces a new Lyapunov function based on quadratic invariants and Hamiltonian dynamics to demonstrate an O(1/k^2) convergence rate for Polyak's Heavy-ball method on convex quadratics, suggesting potential global acceleration.

Contribution

It presents a novel Lyapunov function derived from Hamiltonian invariants, providing the first proof of accelerated convergence for Heavy-ball on convex quadratics.

Findings

Demonstrates O(1/k^2) convergence rate for Heavy-ball on convex quadratics

Introduces a Lyapunov function based on quadratic invariants and Hamiltonian dynamics

Suggests potential for proving global acceleration of Polyak's method

Abstract

In 1964, Polyak showed that the Heavy-ball method, the simplest momentum technique, accelerates convergence of strongly-convex problems in the vicinity of the solution. While Nesterov later developed a globally accelerated version, Polyak's original algorithm remains simpler and more widely used in applications such as deep learning. Despite this popularity, the question of whether Heavy-ball is also globally accelerated or not has not been fully answered yet, and no convincing counterexample has been provided. This is largely due to the difficulty in finding an effective Lyapunov function: indeed, most proofs of Heavy-ball acceleration in the strongly-convex quadratic setting rely on eigenvalue arguments. Our study adopts a different approach: studying momentum through the lens of quadratic invariants of simple harmonic oscillators. By utilizing the modified Hamiltonian of Stormer-Verlet integrators, we are able to construct a Lyapunov function that demonstrates an O(1/k^2) rate for Heavy-ball in the case of convex quadratic problems. This is a promising first step towards potentially proving the acceleration of Polyak's momentum method and we hope it inspires further research in this field.