Numerical Instabilities in the Kaczmarz Method and Stabilization by Iterative Refinement

TL;DR

This paper analyzes the numerical stability issues of randomized Kaczmarz methods and proposes iterative refinement to improve accuracy and stability in solving large-scale linear systems.

Contribution

It reveals the instability of classical and accelerated randomized Kaczmarz methods and introduces iterative refinement as a solution to enhance stability and accuracy.

Findings

Classical and accelerated randomized Kaczmarz are not forward stable.

Iterative refinement effectively controls error accumulation.

Refinement improves solution accuracy even for ill-conditioned systems.

Abstract

The randomized Kaczmarz method and its accelerated variants are a powerful class of iterative methods for solving large-scale linear systems, offering guaranteed convergence with low per-iteration cost. However, their numerical stability remains poorly understood. In this work, we investigate the stability properties of both classical and accelerated randomized Kaczmarz methods, with an emphasis on how error propagates across iterations and interacts with acceleration. We show that both classical and accelerated randomized Kaczmarz fail to be forward stable. To address this issue, we propose the integration of iterative refinement into randomized Kaczmarz frameworks. We demonstrate that refinement can effectively control error accumulation and recover high-accuracy solutions, even when the system is ill-conditioned. Numerical experiments corroborate our theoretical findings and illustrate the practical benefits of combining refinement with both classical and accelerated Kaczmarz methods.