M-IHS: An Accelerated Randomized Preconditioning Method Avoiding Costly Matrix Decompositions

TL;DR

The paper introduces M-IHS, an accelerated randomized preconditioning method for large-scale regularized linear least squares problems that avoids costly matrix decompositions, offering faster convergence and computational savings.

Contribution

It develops the M-IHS algorithm by integrating Heavy Ball acceleration into the Iterative Hessian Sketch, enabling faster convergence without matrix decompositions, and provides theoretical bounds on sketch size.

Findings

M-IHS converges faster than Chebyshev semi-iteration based solvers.

It avoids all matrix decompositions and inversions, reducing computational cost.

The required sketch size is proportional to the statistical dimension, often smaller than the matrix rank.

Abstract

Momentum Iterative Hessian Sketch (M-IHS) techniques, a group of solvers for large scale regularized linear Least Squares (LS) problems, are proposed and analyzed in detail. Proposed M-IHS techniques are obtained by incorporating the Heavy Ball Acceleration into the Iterative Hessian Sketch algorithm and they provide significant improvements over the randomized preconditioning techniques. By using approximate solvers along with the iterations, the proposed techniques are capable of avoiding all matrix decompositions and inversions, which is one of the main advantages over the alternative solvers such as the Blendenpik and the LSRN. Similar to the Chebyshev semi-iterations, the M-IHS variants do not use any inner products and eliminate the corresponding synchronization steps in hierarchical or distributed memory systems, yet the M-IHS converges faster than the Chebyshev Semi-iteration based solvers. Lower bounds on the required sketch size for various randomized distributions are established through the error analyses. Unlike the previously proposed approaches to produce a solution approximation, the proposed M-IHS techniques can use sketch sizes that are proportional to the statistical dimension which is always smaller than the rank of the coefficient matrix. The relative computational saving gets more significant as the regularization parameter or the singular value decay rate of the coefficient matrix increase.