On Block Accelerations of Quantile Randomized Kaczmarz for Corrupted Systems of Linear Equations

TL;DR

This paper introduces an improved block Kaczmarz method for solving large, potentially corrupted linear systems, enhancing robustness and convergence speed by integrating averaging techniques with quantile-based exploration.

Contribution

It proposes a novel approach combining averaged block Kaczmarz with quantile exploration to handle adversarial corruptions more efficiently than existing methods.

Findings

Significantly faster convergence with robust corruption handling.

Theoretical guarantees for convergence under adversarial corruptions.

Classical block Kaczmarz is not robust to sparse adversarial corruptions.

Abstract

With the growth of large data as well as large-scale learning tasks, the need for efficient and robust linear system solvers is greater than ever. The randomized Kaczmarz method (RK) and similar stochastic iterative methods have received considerable recent attention due to their efficient implementation and memory footprint. These methods can tolerate streaming data, accessing only part of the data at a time, and can also approximate the least squares solution even if the system is affected by noise. However, when data is instead affected by large (possibly adversarial) corruptions, these methods fail to converge, as corrupted data points draw iterates far from the true solution. A recently proposed solution to this is the QuantileRK method, which avoids harmful corrupted data by exploring the space carefully as the method iterates. The exploration component requires the computation of quantiles of large samples from the system and is computationally much heavier than the subsequent iteration update. In this paper, we propose an approach that better uses the information obtained during exploration by incorporating an averaged version of the block Kaczmarz method. This significantly speeds up convergence, while still allowing for a constant fraction of the equations to be arbitrarily corrupted. We provide theoretical convergence guarantees as well as experimental supporting evidence. We also demonstrate that the classical projection-based block Kaczmarz method cannot be robust to sparse adversarial corruptions, but rather the blocking has to be carried out by averaging one-dimensional projections.