Convergence directions of the randomized Gauss--Seidel method and its extension

TL;DR

This paper provides a detailed analysis of the convergence behavior of the randomized Gauss--Seidel method and its extension, revealing how singular values influence the convergence directions and rates, and explaining observed empirical phenomena.

Contribution

It offers a novel, detailed convergence analysis that explains the influence of singular values on the convergence directions and rates of these methods.

Findings

Larger singular values lead to faster error decay in corresponding directions.

Convergence directions shift from large to small singular values over iterations.

Numerical experiments confirm the theoretical analysis.

Abstract

The randomized Gauss--Seidel method and its extension have attracted much attention recently and their convergence rates have been considered extensively. However, the convergence rates are usually determined by upper bounds, which cannot fully reflect the actual convergence. In this paper, we make a detailed analysis of their convergence behaviors. The analysis shows that the larger the singular value of $A$ is, the faster the error decays in the corresponding singular vector space, and the convergence directions are mainly driven by the large singular values at the beginning, then gradually driven by the small singular values, and finally by the smallest nonzero singular value. These results explain the phenomenon found in the extensive numerical experiments appearing in the literature that these two methods seem to converge faster at the beginning. Numerical examples are provided to confirm the above findings.