On Subsample Size of Quantile-Based Randomized Kaczmarz

TL;DR

This paper demonstrates that subsampling in quantile-based randomized Kaczmarz methods can significantly reduce computational complexity while maintaining convergence guarantees, especially for large-scale sparse corrupted systems.

Contribution

It introduces a subsampling approach for QRK that reduces the required sample size for quantile computation, providing theoretical convergence guarantees and tight bounds.

Findings

Subsample size D can be as low as C log(T)/log(1/β) for linear convergence.

The proposed method achieves O(log n) complexity for high-accuracy solutions.

Subsample size below a certain threshold leads to arbitrarily large errors.

Abstract

Quantile-based randomized Kaczmarz (QRK) was recently introduced to efficiently solve sparsely corrupted linear systems $\mathbf{A} \mathbf{x}^*+\mathbf{\epsilon} = \mathbf{b}$ [SIAM J. Matrix Anal. Appl., 43(2), 605-637], where $\mathbf{A}\in \mathbb{R}^{m\times n}$ and $\mathbf{\epsilon}$ is an arbitrary $(\beta m)$-sparse corruption. However, all existing theoretical guarantees for QRK require quantiles to be computed using all $m$ samples (or a subsample of the same order), thus negating the computational advantage of Kaczmarz-type methods. This paper overcomes the bottleneck. We analyze a subsampling QRK, which computes quantiles from $D$ uniformly chosen samples at each iteration. Under some standard scaling assumptions on the coefficient matrix, we show that QRK with subsample size $D\ge\frac{C\log (T)}{\log(1/\beta)}$ linearly converges over the first $T$ iterations with high probability, where $C$ is some absolute constant. This subsample size is a substantial reduction from $O(m)$ in prior results. For instance, it translates into $O(\log(n))$ even if an approximation error of $\exp(-n^2)$ is desired. Intriguingly, our subsample size is also tight up to a multiplicative constant: if $D\le \frac{c\log(T)}{\log(1/\beta)}$ for some constant $c$, the error of the $T$-th iterate could be arbitrarily large with high probability. Numerical results are provided to corroborate our theory.