Preconvergence of the randomized extended Kaczmarz method

TL;DR

This paper analyzes the convergence behavior of the randomized extended Kaczmarz method for all linear systems types, revealing that error decay is faster with larger singular values and explaining early convergence phenomena.

Contribution

It provides a comprehensive convergence analysis of the REK method across all linear system types, highlighting the influence of singular values on error decay.

Findings

Error decays faster with larger singular values.

As iterations increase, solutions tend toward the smallest singular value's right singular vector.

Numerical experiments confirm theoretical predictions.

Abstract

In this paper, we analyze the convergence behavior of the randomized extended Kaczmarz (REK) method for all types of linear systems (consistent or inconsistent, overdetermined or underdetermined, full-rank or rank-deficient). The analysis shows that the larger the singular value of $A$ is, the faster the error decays in the corresponding right singular vector space, and as $k\rightarrow\infty$, $x_{k}-x_{\star}$ tends to the right singular vector corresponding to the smallest singular value of $A$, where $x_{k}$ is the $k$th approximation of the REK method and $x_{\star}$ is the minimum $\ell_2 $-norm least squares solution. These results explain the phenomenon found in the extensive numerical experiments appearing in the literature that the REK method seems to converge faster in the beginning. A simple numerical example is provided to confirm the above findings.