Randomized Extended Kaczmarz is a Limit Point of Sketch-and-Project

TL;DR

This paper demonstrates that the randomized extended Kaczmarz method can be viewed as a limit point of the sketch-and-project framework, providing new theoretical insights and practical connections.

Contribution

It shows that REK is a limit point of SAP methods and offers a detailed theoretical analysis and experimental validation of this relationship.

Findings

REK can be expressed as a limit point of SAP methods

Theoretical convergence guarantees for the related SAP family

Experimental evidence supporting the connection between REK and SAP

Abstract

The sketch-and-project (SAP) framework for solving systems of linear equations has unified the theory behind popular projective iterative methods such as randomized Kaczmarz, randomized coordinate descent, and variants thereof. The randomized extended Kaczmarz (REK) method is a popular extension of randomized Kaczmarz for solving inconsistent systems, which has not yet been shown to lie within the SAP framework. In this work we show that, in a certain sense, REK may be expressed as the limit point of a family of SAP methods, but we argue that it is unlikely that REK can be translated into a SAP method itself. We provide an extensive theoretical analysis of the family of methods comprising said limit, including convergence guarantees and further connections to REK. We follow this with an array of experiments demonstrating these methods and their connections in practice.