On Adapting Nesterov's Scheme to Accelerate Iterative Methods for Linear Problems

TL;DR

This paper adapts Nesterov's acceleration scheme to iterative methods for linear systems, providing explicit optimal parameters, demonstrating robustness over complex eigenvalues, and validating efficiency through numerical tests.

Contribution

It introduces a Nesterov-inspired acceleration scheme for linear solvers, with explicit optimal parameters and robustness analysis for complex eigenvalues.

Findings

The scheme requires more iterations than classical methods but is easy to implement.

Explicit formulas for optimal parameters are derived based on eigenvalues.

Numerical tests confirm the scheme's efficiency and robustness.

Abstract

Nesterov's well-known scheme for accelerating gradient descent in convex optimization problems is adapted to accelerating stationary iterative solvers for linear systems. Compared with classical Krylov subspace acceleration methods, the proposed scheme requires more iterations, but it is trivial to implement and retains essentially the same computational cost as the unaccelerated method. An explicit formula for a fixed optimal parameter is derived in the case where the stationary iteration matrix has only real eigenvalues, based only on the smallest and largest eigenvalues. The fixed parameter, and corresponding convergence factor, are shown to maintain their optimality when the iteration matrix also has complex eigenvalues that are contained within an explicitly defined disk in the complex plane. A comparison to Chebyshev acceleration based on the same information of the smallest and largest real eigenvalues (dubbed Restricted Information Chebyshev acceleration) demonstrates that Nesterov's scheme is more robust in the sense that it remains optimal over a larger domain when the iteration matrix does have some complex eigenvalues. Numerical tests validate the efficiency of the proposed scheme. This work generalizes and extends the results of [1, Lemmas 3.1 and 3.2 and Theorem 3.3].