Adaptively Sketched Bregman Projection Methods for Linear Systems

TL;DR

This paper introduces the sketched Bregman projection (SBP) method, a flexible framework for solving linear systems that encompasses existing methods and allows for solutions with specific structures, supported by theoretical convergence and numerical experiments.

Contribution

It proposes the SBP method, generalizing adaptive sampling in linear system solvers, and provides convergence analysis and empirical validation for structured solutions.

Findings

SBP method unifies various iterative algorithms for linear systems.

Adaptive sampling rules improve convergence efficiency.

Numerical results show sampling Kaczmarz-Motzkin rule reduces computational costs.

Abstract

The sketch-and-project, as a general archetypal algorithm for solving linear systems, unifies a variety of randomized iterative methods such as the randomized Kaczmarz and randomized coordinate descent. However, since it aims to find a least-norm solution from a linear system, the randomized sparse Kaczmarz can not be included. This motivates us to propose a more general framework, called sketched Bregman projection (SBP) method, in which we are able to find solutions with certain structures from linear systems. To generalize the concept of adaptive sampling to the SBP method, we show how the progress, measured by Bregman distance, of single step depends directly on a sketched loss function. Theoretically, we provide detailed global convergence results for the SBP method with different adaptive sampling rules. At last, for the (sparse) Kaczmarz methods, a group of numerical simulations are tested, with which we verify that the methods utilizing sampling Kaczmarz-Motzkin rule demands the fewest computational costs to achieve a given error bound comparing to the corresponding methods with other sampling rules.