Quadratic minimization: from conjugate gradient to an adaptive Heavy-ball method with Polyak step-sizes

TL;DR

This paper introduces an adaptive Heavy-ball method using Polyak step-sizes for convex quadratic minimization, connecting it to conjugate gradient methods and analyzing the impact of momentum on convergence.

Contribution

It presents a novel adaptive Heavy-ball method with Polyak step-sizes, establishing its equivalence to conjugate gradient and analyzing momentum effects in quadratic minimization.

Findings

The method inherits conjugate gradient's finite-time convergence.

It achieves instance optimality and known worst-case rates.

Incorporating momentum can improve convergence in quadratic problems.

Abstract

In this work, we propose an adaptive variation on the classical Heavy-ball method for convex quadratic minimization. The adaptivity crucially relies on so-called "Polyak step-sizes", which consists in using the knowledge of the optimal value of the optimization problem at hand instead of problem parameters such as a few eigenvalues of the Hessian of the problem. This method happens to also be equivalent to a variation of the classical conjugate gradient method, and thereby inherits many of its attractive features, including its finite-time convergence, instance optimality, and its worst-case convergence rates. The classical gradient method with Polyak step-sizes is known to behave very well in situations in which it can be used, and the question of whether incorporating momentum in this method is possible and can improve the method itself appeared to be open. We provide a definitive answer to this question for minimizing convex quadratic functions, a arguably necessary first step for developing such methods in more general setups.