Eigenvector-based acceleration strategies for gradient-type methods

TL;DR

This paper introduces eigenvector-based acceleration techniques for gradient methods that improve convergence by relaxing step lengths and leveraging Lanczos method properties to avoid zigzagging and approach eigenvectors.

Contribution

It proposes novel strategies to accelerate gradient methods by relaxing step lengths and utilizing eigenvector approximations via Lanczos, enhancing convergence for convex functions.

Findings

Strategies effectively reduce iteration counts

Approach mitigates zigzagging effect

Accelerates convergence towards the minimizer

Abstract

Several strategies are described and analyzed to speed-up gradient-type methods when applied to the minimization of strictly convex quadratics and strictly convex functions. The proposed techniques focus on relaxing the traditional optimal step length associated with gradient methods, including the steepest descent (SD) and the minimal residual (MR) methods. Such a relaxation avoids the well-known negative zigzag effect and allows the iterates to move in the entire space which in turn implies that every so often the search direction approaches some eigenvector of the underlying Hessian matrix. The proposed speedups then rely on taking advantage of the properties of the Lanczos method once a search direction that approaches an eigenvector has been identified in order to accelerate the convergence towards the global minimizer. After analyzing the proposed strategies, we illustrate them on the global minimization of strictly convex functions.