Randomized Kaczmarz with geometrically smoothed momentum

TL;DR

This paper investigates how geometrically smoothed momentum enhances the randomized Kaczmarz algorithm, providing theoretical error bounds and numerical evidence of improved performance in solving linear least squares problems.

Contribution

It introduces the incorporation of geometrically smoothed momentum into the randomized Kaczmarz algorithm and analyzes its effect on expected error in singular vector directions.

Findings

Theoretical bounds on expected error with smoothed momentum

Numerical examples demonstrating improved convergence

Open questions for future research

Abstract

This paper studies the effect of adding geometrically smoothed momentum to the randomized Kaczmarz algorithm, which is an instance of stochastic gradient descent on a linear least squares loss function. We prove a result about the expected error in the direction of singular vectors of the matrix defining the least squares loss. We present several numerical examples illustrating the utility of our result and pose several questions.