Kaczmarz Kac Walk

TL;DR

This paper introduces a novel iterative method inspired by the Kaczmarz algorithm, involving projections between random hyperplanes, which accelerates the growth of small singular values while preserving the original solution.

Contribution

It proposes a new hyperplane projection technique that modifies the linear system to enhance convergence properties and spectral characteristics.

Findings

Small singular values grow exponentially with iterations

Method preserves the original solution

Accelerates convergence of linear system solutions

Abstract

The Kaczmarz method is a way to iteratively solve a linear system of equations $Ax = b$. One interprets the solution $x$ as the point where hyperplanes intersect and then iteratively projects an approximate solution onto these hyperplanes to get better and better approximations. We note a somewhat related idea: one could take two random hyperplanes and project one into the orthogonal complement of the other. This leads to a sequence of linear systems $A^{(k)} x = b^{(k)}$ which is fast to compute, preserves the original solution and whose small singular values grow like $\sigma_{\ell}(A^{(k)}) \sim \exp(k/n^2) \cdot \sigma_{\ell}(A)$.