Fine-grained Analysis and Faster Algorithms for Iteratively Solving Linear Systems

TL;DR

This paper introduces a fine-grained complexity measure for solving large linear systems, proposing a stochastic algorithm that improves efficiency especially for data with low-dimensional structure, outperforming traditional methods like conjugate gradient.

Contribution

The paper develops a new stochastic Sketch-and-Project algorithm with improved runtime bounds based on a fine-grained spectral ratio, advancing linear system solving in structured data scenarios.

Findings

Achieves solution in time $ar O(ppa_ll \, n^2 \log 1/\epsilon)$ for certain spectral ratios.

Provides a new analysis of the moments of the random projection matrix in Sketch-and-Project.

Demonstrates a separation between stochastic solvers and traditional matrix-vector product algorithms.

Abstract

Despite being a key bottleneck in many machine learning tasks, the cost of solving large linear systems has proven challenging to quantify due to problem-dependent quantities such as condition numbers. To tackle this, we consider a fine-grained notion of complexity for solving linear systems, which is motivated by applications where the data exhibits low-dimensional structure, including spiked covariance models and kernel machines, and when the linear system is explicitly regularized, such as ridge regression. Concretely, let $\kappa_\ell$ be the ratio between the $\ell$th largest and the smallest singular value of $n\times n$ matrix $A$. We give a stochastic algorithm based on the Sketch-and-Project paradigm, that solves the linear system $Ax = b$, that is, finds $\bar{x}$ such that $\|A\bar{x} - b\| \le \epsilon \|b\|$, in time $\bar O(\kappa_\ell\cdot n^2\log 1/\epsilon)$, for any $\ell = O(n^{0.729})$. This is a direct improvement over preconditioned conjugate gradient, and it provides a stronger separation between stochastic linear solvers and algorithms accessing $A$ only through matrix-vector products. Our main technical contribution is the new analysis of the first and second moments of the random projection matrix that arises in Sketch-and-Project.