A Numerical Analysis of Sketched Linear Squares Problems and Stopping Criteria for Iterative Solvers

TL;DR

This paper provides a detailed numerical analysis of sketched linear least squares problems, establishing theoretical bounds and proposing new stopping criteria for iterative solvers that significantly reduce computational costs.

Contribution

It introduces novel theoretical bounds, analyzes the backward error of sketched LS solutions, and develops reliable stopping criteria for iterative solvers based on stabilization patterns.

Findings

The residual difference norm between LS and sLS solutions is tightly bounded.

The sLS solution is the minimal backward error solution of a perturbed LS problem.

New stopping criteria reduce computational cost without losing accuracy.

Abstract

Randomized subspace embedding methods have had a great impact on the solution of a linear least squares (LS) problem by reducing its row dimension, leading to a randomized or sketched LS (sLS) problem, and use the solution of the sLS problem as an approximate solution of the LS problem. This work makes a numerical analysis on the sLS problem, establishes its numerous theoretical properties, and show their crucial roles on the most effective and efficient use of iterative solvers. We first establish a compact bound on the norm of the residual difference between the solutions of the LS and sLS problems, which is the first key result towards understanding the rationale of the sLS problem. Then from the perspective of backward errors, we prove that the solution of the sLS problem is the one of a certain perturbed LS problem with minimal backward error, and quantify how the embedded quality affects the residuals, solution errors, and the relative residual norms of normal equations of the LS and sLS problems. These theoretical results enable us to propose new novel and reliable general-purpose stopping criteria for iterative solvers for the sLS problem, which dynamically monitor stabilization patterns of iterative solvers for the LS problem itself and terminate them at the earliest iteration. Numerical experiments justify the theoretical bounds and demonstrate that the new stopping criteria work reliably and result in a tremendous reduction in computational cost without sacrificing attainable accuracy.