Kaczmarz and Gauss-Seidel Algorithms with Volume Sampling

TL;DR

This paper extends the analysis of the Alternating Projections method, including Kaczmarz and Gauss-Seidel algorithms, by incorporating volume sampling for larger block sizes, showing improved convergence rates related to the system spectrum.

Contribution

It introduces volume sampling into the analysis of block randomized AP methods, providing explicit convergence rate formulas and demonstrating monotonic improvements with larger block sizes.

Findings

Convergence rates improve with larger block sizes.

Explicit formulas relate convergence to the spectrum of the system.

Volume sampling outperforms uniform sampling in convergence guarantees.

Abstract

The method of Alternating Projections (AP) is a fundamental iterative technique with applications to problems in machine learning, optimization and signal processing. Examples include the Gauss-Seidel algorithm which is used to solve large-scale regression problems and the Kaczmarz and projections onto convex sets (POCS) algorithms that are fundamental to iterative reconstruction. Progress has been made with regards to the questions of efficiency and rate of convergence in the randomized setting of the AP method. Here, we extend these results with volume sampling to block (batch) sizes greater than 1 and provide explicit formulas that relate the convergence rate bounds to the spectrum of the underlying system. These results, together with a trace formula and associated volume sampling, prove that convergence rates monotonically improve with larger block sizes, a feature that can not be guaranteed in general with uniform sampling (e.g., in SGD).