Beyond Expectation: Concentration Inequalities for Randomized Iterative Methods

TL;DR

This paper develops concentration inequalities and variance bounds for a broad class of stochastic iterative methods, enhancing understanding of their near-worst-case behavior beyond traditional expected error analysis.

Contribution

It introduces new theoretical bounds on the concentration and variance of errors for both linear and nonlinear stochastic iterative methods, including randomized Kaczmarz and Gauss-Seidel.

Findings

Provides upper bounds for error concentration and variance.

Applies to linear and nonlinear stochastic iterative methods.

Enhances understanding of near-worst-case behavior.

Abstract

Stochastic iterative methods are useful in a variety of large-scale numerical linear algebraic, machine learning, and statistical problems, in part due to their low-memory footprint. They are frequently used in a variety of applications, and thus it is imperative to have a thorough theoretical understanding of their behavior. Most theoretical convergence results for stochastic iterative methods provide bounds on the expected error of the iterates, and yield a type of average case analysis. However, understanding the behavior of these methods in the near-worst-case is desirable. For stochastic methods, this motivates providing bounds on the variance and concentration of their error, which can be used to generate confidence intervals around the bounds on their expected error. Here, we provide upper bounds for the concentration and variance of the error of a general class of linear stochastic iterative methods, including the randomized Kaczmarz method and the randomized Gauss--Seidel method, and a more general class of nonlinear stochastic iterative methods, including the randomized Kaczmarz method for systems of linear inequalities.