Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and $\lambda$-Effectiveness

TL;DR

This paper develops and analyzes infinite-dimensional Kaczmarz algorithms with relaxation parameters, providing explicit regret bounds and demonstrating their effectiveness in machine learning models, including noisy data scenarios.

Contribution

It introduces a comprehensive framework for infinite-dimensional Kaczmarz algorithms with explicit regret bounds and detailed analysis of relaxation parameters, advancing their application in modern machine learning.

Findings

Explicit regret bounds for Kaczmarz algorithms in infinite-dimensional spaces

Analysis of the impact of relaxation parameters on algorithm performance

Application of regret estimates to noisy data in machine learning

Abstract

We present a variety of projection-based linear regression algorithms with a focus on modern machine-learning models and their algorithmic performance. We study the role of the relaxation parameter in generalized Kaczmarz algorithms and establish a priori regret bounds with explicit $\lambda$-dependence to quantify how much an algorithm's performance deviates from its optimal performance. A detailed analysis of relaxation parameter is also provided. Applications include: explicit regret bounds for the framework of Kaczmarz algorithm models, non-orthogonal Fourier expansions, and the use of regret estimates in modern machine learning models, including for noisy data, i.e., regret bounds for the noisy Kaczmarz algorithms. Motivated by machine-learning practice, our wider framework treats bounded operators (on infinite-dimensional Hilbert spaces), with updates realized as (block) Kaczmarz algorithms, leading to new and versatile results.