On the convergence analysis of the greedy randomized Kaczmarz method

TL;DR

This paper provides a detailed convergence analysis of the greedy randomized Kaczmarz method, demonstrating deterministic linear convergence and improving iteration complexity, with enhancements via Polyak's momentum technique.

Contribution

It introduces more precise greedy probability criteria, revises existing convergence analyses to show deterministic convergence, and incorporates momentum to enhance performance.

Findings

Proves deterministic linear convergence of the GRK method.

Shows that a tighter threshold parameter accelerates convergence.

Demonstrates that momentum improves the efficiency of the GRK method.

Abstract

In this paper, we analyze the greedy randomized Kaczmarz (GRK) method proposed in Bai and Wu (SIAM J. Sci. Comput., 40(1):A592--A606, 2018) for solving linear systems. We develop more precise greedy probability criteria to effectively select the working row from the coefficient matrix. Notably, we prove that the linear convergence of the GRK method is deterministic and demonstrate that using a tighter threshold parameter can lead to a faster convergence rate. Our result revises existing convergence analyses, which are solely based on the expected error by realizing that the iterates of the GRK method are random variables. Consequently, we obtain an improved iteration complexity for the GRK method. Moreover, the Polyak's heavy ball momentum technique is incorporated to improve the performance of the GRK method. We propose a refined convergence analysis, compared with the technique used in Loizou and Richt\'{a}rik (Comput. Optim. Appl., 77(3):653--710, 2020), of momentum variants of randomized iterative methods, which shows that the proposed GRK method with momentum (mGRK) also enjoys a deterministic linear convergence. Numerical experiments show that the mGRK method is more efficient than the GRK method.