Randomized Kaczmarz with tail averaging

TL;DR

This paper introduces Tail-Averaged Randomized Kaczmarz (TARK), an improved algorithm that guarantees convergence for both consistent and inconsistent linear systems by averaging tail iterates, achieving optimal polynomial rates.

Contribution

The paper proposes a simple tail-averaging modification to RK that ensures convergence in inconsistent systems and extends to regularized least-squares problems.

Findings

TARK converges for inconsistent systems at an optimal polynomial rate.

Tail averaging improves RK's convergence to the least-squares solution.

Extension to ridge-regularized problems is effective.

Abstract

The randomized Kaczmarz (RK) method is a well-known approach for solving linear least-squares problems with a large number of rows. RK accesses and processes just one row at a time, leading to exponentially fast convergence for consistent linear systems. However, RK fails to converge to the least-squares solution for inconsistent systems. This work presents a simple fix: average the RK iterates produced in the tail part of the algorithm. The proposed tail-averaged randomized Kaczmarz (TARK) converges for both consistent and inconsistent least-squares problems at a polynomial rate, which is known to be optimal for any row-access method. An extension of TARK also leads to efficient solutions for ridge-regularized least-squares problems.