Polynomial Preconditioning for Gradient Methods

TL;DR

This paper introduces a novel polynomial preconditioning technique for gradient methods that improves convergence by better conditioning, applicable without explicit spectral knowledge, and demonstrates efficiency in machine learning tasks.

Contribution

It proposes a new family of polynomial preconditioners based on symmetric polynomials, enhancing gradient methods with provable condition number improvements and adaptive selection strategies.

Findings

Improved convergence rates for gradient methods using polynomial preconditioning.

Effective automatic selection of optimal polynomial preconditioners.

Numerical experiments show significant efficiency gains in machine learning applications.

Abstract

We study first-order methods with preconditioning for solving structured nonlinear convex optimization problems. We propose a new family of preconditioners generated by symmetric polynomials. They provide first-order optimization methods with a provable improvement of the condition number, cutting the gaps between highest eigenvalues, without explicit knowledge of the actual spectrum. We give a stochastic interpretation of this preconditioning in terms of coordinate volume sampling and compare it with other classical approaches, including the Chebyshev polynomials. We show how to incorporate a polynomial preconditioning into the Gradient and Fast Gradient Methods and establish the corresponding global complexity bounds. Finally, we propose a simple adaptive search procedure that automatically chooses the best possible polynomial preconditioning for the Gradient Method, minimizing the objective along a low-dimensional Krylov subspace. Numerical experiments confirm the efficiency of our preconditioning strategies for solving various machine learning problems.