Randomized Kaczmarz converges along small singular vectors

TL;DR

This paper reveals that the Randomized Kaczmarz method converges to solutions along the smallest singular vectors of the matrix, providing insights into its convergence behavior and enabling efficient computation of vectors with small Rayleigh quotients.

Contribution

It demonstrates that the sequence generated by Randomized Kaczmarz converges to the solution along small singular vectors, offering a new perspective on its convergence and error analysis.

Findings

Convergence to small singular vectors as iterations increase

Optimality of existing error bounds

Fast computation of vectors with small Rayleigh quotient

Abstract

Randomized Kaczmarz is a simple iterative method for finding solutions of linear systems $Ax = b$. We point out that the arising sequence $(x_k)_{k=1}^{\infty}$ tends to converge to the solution $x$ in an interesting way: generically, as $k \rightarrow \infty$, $x_k - x$ tends to the singular vector of $A$ corresponding to the smallest singular value. This has interesting consequences: in particular, the error analysis of Strohmer \& Vershynin is optimal. It also quantifies the `pre-convergence' phenomenon where the method initially seems to converge faster. This fact also allows for a fast computation of vectors $x$ for which the Rayleigh quotient $\|Ax\|/\|x\|$ is small: solve $Ax = 0$ via Randomized Kaczmarz.