Adaptive Momentum via Minimal Dual Function for Accelerating Randomized Sparse Kaczmarz

TL;DR

This paper introduces a new adaptive momentum method based on the minimal dual function for accelerated randomized sparse Kaczmarz, improving convergence and robustness for various linear systems.

Contribution

The paper develops a novel adaptive momentum technique using the minimal dual function, extending acceleration beyond exact measurements and integrating it with quantile-based sampling.

Findings

Achieves linear convergence in expectation up to a contamination-dependent horizon.

Effectively handles exact, noisy, and corrupted linear systems.

Demonstrates improved performance on simulated and real data.

Abstract

Recently, the randomized sparse Kaczmarz method has been accelerated by designing heavy ball momentum adaptively via a minimal-error principle. In this paper, we develop a new adaptive momentum method based on the minimal dual function principle to go beyond the exact measurement restriction of the minimal-error principle. Moreover, by integrating the new adaptive momentum method with the quantile-based sampling, we introduce a general algorithmic framework, called quantile-based randomized sparse Kaczmarz with minimal dual function momentum, which provides a unified approach to exact, noisy, or corrupted linear systems. In addition, we utilize the discrepancy principle and monotone error as stopping rules for the proposed algorithm. Theoretically, we establish linear convergence in expectation of Bregman distance up to a finite horizon related to the contaminated level. At last, we provide numerical illustrations on simulated and real-world data to demonstrate the effectiveness of our proposed method.