The conjugate gradient algorithm on a general class of spiked covariance matrices

TL;DR

This paper analyzes the conjugate gradient algorithm's behavior on spiked covariance matrices, revealing universal concentration properties and showing that classical bounds are overly pessimistic for such matrices.

Contribution

It establishes universal concentration results for the error and residual norms of the conjugate gradient algorithm on spiked covariance matrices, extending understanding of its convergence behavior.

Findings

Error and residual norms concentrate on deterministic values

Universal limits and fluctuations are established

Classical condition number bounds are overly pessimistic for spiked matrices

Abstract

We consider the conjugate gradient algorithm applied to a general class of spiked sample covariance matrices. The main result of the paper is that the norms of the error and residual vectors at any finite step concentrate on deterministic values determined by orthogonal polynomials with respect to a deformed Marchenko--Pastur law. The first-order limits and fluctuations are shown to be universal. Additionally, for the case where the bulk eigenvalues lie in a single interval we show a stronger universality result in that the asymptotic rate of convergence of the conjugate gradient algorithm only depends on the support of the bulk, provided the spikes are well-separated from the bulk. In particular, this shows that the classical condition number bound for the conjugate gradient algorithm is pessimistic for spiked matrices.