The matrix-vector complexity of $Ax=b$

TL;DR

This paper establishes fundamental lower bounds on the number of matrix-vector products required by Krylov subspace methods to solve large linear systems, confirming their optimality and limitations.

Contribution

It provides the first rigorous lower bounds on matrix-vector products needed for approximate solutions, matching known upper bounds and demonstrating the optimality of Krylov methods.

Findings

Omega(kappa log(1/epsilon)) products needed with randomization and transpose access

One-sided algorithms require n products for n x n systems

Results include explicit constants matching upper bounds up to a factor of four

Abstract

Matrix--vector algorithms, particularly Krylov subspace methods, are widely viewed as the most effective algorithms for solving large systems of linear equations. This paper establishes lower bounds on the worst-case number of matrix--vector products needed by such an algorithm to approximately solve a general linear system. The first main result is that, for any matrix--vector algorithm which is allowed the use of randomization and can perform products with both a matrix and its transpose, $\Omega(\kappa \log(1/\varepsilon))$ matrix--vector products are necessary to solve a linear system with condition number $\kappa$ to accuracy $\varepsilon$, matching an upper bound for conjugate gradient on the normal equations. The second main result is that one-sided algorithms, which lack access to the transpose, must use $n$ matrix--vector products to solve an $n \times n$ linear system, even when the problem is perfectly conditioned. Both main results include explicit constants that match known upper bounds up to a factor of four. These results rigorously demonstrate the limitations of matrix--vector algorithms and confirm the optimality of widely used Krylov subspace algorithms.