An optimal scheduled learning rate for a randomized Kaczmarz algorithm

TL;DR

This paper develops an optimal learning rate schedule for a relaxed randomized Kaczmarz algorithm, improving convergence bounds for solving noisy linear systems by leveraging a sharp bound involving the Lambert-W function.

Contribution

It introduces a novel learning rate schedule that optimizes the expected error bound for the randomized Kaczmarz algorithm in noisy settings.

Findings

The optimized schedule outperforms standard approaches in convergence speed.

The bound involves the Lambert-W function, providing a new analytical perspective.

Experimental results confirm improved performance with the proposed schedule.

Abstract

We study how the learning rate affects the performance of a relaxed randomized Kaczmarz algorithm for solving $A x \approx b + \varepsilon$, where $A x =b$ is a consistent linear system and $\varepsilon$ has independent mean zero random entries. We derive a learning rate schedule which optimizes a bound on the expected error that is sharp in certain cases; in contrast to the exponential convergence of the standard randomized Kaczmarz algorithm, our optimized bound involves the reciprocal of the Lambert-$W$ function of an exponential.