Randomized batch-sampling Kaczmarz methods for solving linear systems

TL;DR

This paper introduces a unified randomized batch-sampling Kaczmarz framework for solving linear systems, providing new convergence guarantees and tighter bounds that improve understanding of block methods.

Contribution

It develops a general analysis technique for randomized block Kaczmarz methods, deriving scale-invariant convergence bounds applicable to arbitrary stochastic samplings.

Findings

New expected linear convergence rate bounds are derived.

Bounds are tighter and more reflective of empirical behavior.

The framework allows for learnable batch-sampling distributions.

Abstract

To conduct a more in-depth investigation of randomized solvers for solving linear systems, we adopt a unified randomized batch-sampling Kaczmarz framework with per-iteration costs as low as cyclic block methods, and develop a general analysis technique to establish its convergence guarantee. With concentration inequalities, we derive new expected linear convergence rate bounds. The analysis applies to any randomized non-extended block Kaczmarz methods with arbitrary static stochastic samplings. In addition, the new rate bounds are scale-invariant, which eliminate the dependence on the magnitude of the data matrix. In most experiments, the new bounds are significantly tighter than existing ones and better reflect the empirical convergence behavior of block methods. Within this new framework, the batch-sampling distribution, as a learnable parameter, provides the possibility for block methods to achieve efficient performance in specific application scenarios, which deserves further investigation.