Randomized double and triple Kaczmarz for solving extended normal equations

TL;DR

This paper introduces randomized double and triple Kaczmarz algorithms for efficiently solving extended normal equations without explicitly forming the normal matrix, applicable to any matrix A, with proven exponential convergence.

Contribution

The paper proposes novel randomized double and triple Kaczmarz algorithms that handle extended normal equations for any matrix A, avoiding explicit matrix formation and providing theoretical convergence guarantees.

Findings

Algorithms achieve exponential convergence in mean square sense.

Numerical experiments confirm theoretical convergence rates.

Abstract

The randomized Kaczmarz algorithm has received considerable attention recently because of its simplicity, speed, and the ability to approximately solve large-scale linear systems of equations. In this paper we propose randomized double and triple Kaczmarz algorithms to solve extended normal equations of the form $\bf A^\top Ax=A^\top b-c$. The proposed algorithms avoid forming $\bf A^\top A$ explicitly and work for {\it arbitrary} $\mbf A\in\mbbr^{m\times n}$ (full rank or rank deficient, $m\geq n$ or $m<n$). {\it Tight} upper bounds showing exponential convergence in the mean square sense of the proposed algorithms are presented and numerical experiments are given to illustrate the theoretical results.