Approaching Optimality for Solving Dense Linear Systems with Low-Rank Structure

TL;DR

This paper introduces new randomized algorithms for solving dense linear systems with low-rank structure efficiently, nearly matching the theoretical complexity limits and improving upon previous methods in terms of speed and accuracy.

Contribution

The paper presents the first nearly-linear time algorithms for solving certain low-rank structured linear systems and approximating the nuclear norm of dense matrices.

Findings

Algorithms succeed with high probability under well-conditioned assumptions.

Running times are nearly optimal, improving previous trade-offs.

First nearly-linear time algorithm for nuclear norm approximation.

Abstract

We provide new high-accuracy randomized algorithms for solving linear systems and regression problems that are well-conditioned except for $k$ large singular values. For solving such $d \times d$ positive definite system our algorithms succeed whp. and run in time $\tilde O(d^2 + k^\omega)$. For solving such regression problems in a matrix $\mathbf{A} \in \mathbb{R}^{n \times d}$ our methods succeed whp. and run in time $\tilde O(\mathrm{nnz}(\mathbf{A}) + d^2 + k^\omega)$ where $\omega$ is the matrix multiplication exponent and $\mathrm{nnz}(\mathbf{A})$ is the number of non-zeros in $\mathbf{A}$. Our methods nearly-match a natural complexity limit under dense inputs for these problems and improve upon a trade-off in prior approaches that obtain running times of either $\tilde O(d^{2.065}+k^\omega)$ or $\tilde O(d^2 + dk^{\omega-1})$ for $d\times d$ systems. Moreover, we show how to obtain these running times even under the weaker assumption that all but $k$ of the singular values have a suitably bounded generalized mean. Consequently, we give the first nearly-linear time algorithm for computing a multiplicative approximation to the nuclear norm of an arbitrary dense matrix. Our algorithms are built on three general recursive preconditioning frameworks, where matrix sketching and low-rank update formulas are carefully tailored to the problems' structure.