Linear convergence of Gearhart-Koshy accelerated Kaczmarz methods for tensor linear systems

TL;DR

This paper proves the linear convergence of the Gearhart-Koshy accelerated Kaczmarz method for tensor linear systems, applicable to various variants, with improved convergence bounds and efficient implementation.

Contribution

It establishes the first rigorous proof of linear convergence for the generalized Gearhart-Koshy acceleration in tensor Kaczmarz methods, including multiple variants.

Findings

The method converges linearly to the least-norm solution.

The acceleration improves convergence bounds over plain Kaczmarz.

An efficient Gram-Schmidt-based implementation computes iterates in linear time.

Abstract

The generalized Gearhart-Koshy acceleration is a recent exact affine search technique designed for the method of cyclic projections onto hyperplanes, i.e., the Kaczmarz method. However, its convergence properties, particularly the linear convergence rate, have not been thoroughly established. In this paper, we systematically establish the linear convergence of the generalized Gearhart-Koshy accelerated Kaczmarz method for tensor linear systems, proving that it converges linearly to the unique least-norm solution. Our analysis is general and applies to several popular Kaczmarz variants, including incremental, shuffle-once, and random-reshuffling schemes, and demonstrates that this acceleration approach yields a better convergence upper bound compared to the plain Kaczmarz method. We also propose an efficient Gram-Schmidt-based implementation that computes the next iterate in linear time. Building on this implementation, we establish a connection between this acceleration framework and Arnoldi-type Krylov subspace methods, further highlighting its efficiency and potential. Our theoretical results are supported by numerical experiments.