Beyond Low Rank: Fast Low-Rank + Diagonal Decomposition with a Spectral Approach

TL;DR

This paper introduces a fast, scalable spectral approach for low-rank plus diagonal matrix decomposition, improving accuracy and efficiency over existing methods for large covariance matrices.

Contribution

It proposes a novel alternating algorithm and a randomized hybrid method with theoretical error bounds for low-rank plus diagonal decompositions.

Findings

Achieves machine precision error with fewer matrix-vector products

Outperforms gradient descent and majorization-minimization methods

Successfully recovers structured matrices and improves covariance approximation in real data

Abstract

Low-rank plus diagonal (LRPD) decompositions provide a powerful structural model for large covariance matrices, simultaneously capturing global shared factors and localized corrections that arise in covariance estimation, factor analysis, and large-scale kernel learning. We introduce an alternating low-rank then diagonal (Alt) algorithm that provably reduces approximation error and significantly outperforms gradient descent while remaining cheaper than majorization-minimization methods~\cite{sun2016majorization}. To scale to large matrices, we develop a randomized LRPD variant that combines fixed-rank Nystrom sketching~\cite{tropp2017fixed} for the low-rank component with Diag++ stochastic diagonal estimation~\cite{baston2022stochastic}. This hybrid algorithm achieves machine precision decomposition error using a number of matrix-vector products far smaller than the ambient dimension, and comes with rigorous non-asymptotic error bounds. On synthetic data, it exactly recovers LRPD structured matrices with high efficiency, and on real-world S&P 500 stock return covariances, where the spectrum decays slowly and strong sector structure exists, it achieves substantially lower error than pure low-rank approximations.