Mixed Precision $s$-step Lanczos and Conjugate Gradient Algorithms

TL;DR

This paper introduces a mixed precision approach to $s$-step Lanczos and Conjugate Gradient algorithms, significantly improving their numerical stability and accuracy by performing select computations in higher precision, thus enabling better performance without major overhead.

Contribution

The paper demonstrates that performing select computations in double precision reduces error amplification in $s$-step algorithms, enhancing their numerical stability and extending their practical applicability.

Findings

Mixed precision reduces error amplification in $s$-step algorithms.

Numerical experiments confirm improved stability and accuracy.

Extended to $s$-step CG with similar benefits.

Abstract

Compared to the classical Lanczos algorithm, the $s$-step Lanczos variant has the potential to improve performance by asymptotically decreasing the synchronization cost per iteration. However, this comes at a cost. Despite being mathematically equivalent, the $s$-step variant is known to behave quite differently in finite precision, with potential for greater loss of accuracy and a decrease in the convergence rate relative to the classical algorithm. It has previously been shown that the errors that occur in the $s$-step version follow the same structure as the errors in the classical algorithm, but with the addition of an amplification factor that depends on the square of the condition number of the $O(s)-$dimensional Krylov bases computed in each outer loop. As the condition number of these $s$-step bases grows (in some cases very quickly) with $s$, this limits the parameter $s$ that can be chosen and thus limits the performance that can be achieved. In this work we show that if a select few computations in $s$-step Lanczos are performed in double the working precision, the error terms then depend only linearly on the conditioning of the $s$-step bases. This has the potential for drastically improving the numerical behavior of the algorithm with little impact on per-iteration performance. Our numerical experiments demonstrate the improved numerical behavior possible with the mixed precision approach, and also show that this improved behavior extends to the $s$-step CG algorithm in mixed precision.