Read Between the Hyperplanes: On Spectral Projection and Sampling Approaches to Randomized Kaczmarz

TL;DR

This paper explores spectral projection and sampling strategies to enhance the convergence of the Randomized Kaczmarz method for large, ill-conditioned linear systems by leveraging inter-row relationships.

Contribution

It introduces novel projection and sampling techniques based on spectral properties and row relationships to accelerate convergence in RK.

Findings

Projection onto pairwise row differences improves convergence.

Sampling from nearly orthogonal row clusters enhances efficiency.

Spectrally-diverse row sampling leads to faster convergence.

Abstract

Among recent developments centered around Randomized Kaczmarz (RK), a row-sampling iterative projection method for large-scale linear systems, several adaptions to the method have inspired faster convergence. Focusing solely on ill-conditioned and overdetermined linear systems, we highlight inter-row relationships that can be leveraged to guide directionally aware projections. In particular, we find that improved convergence rates can be made by (i) projecting onto pairwise row differences, (ii) sampling from partitioned clusters of nearly orthogonal rows, or (iii) more frequently sampling spectrally-diverse rows.