Spectral Estimation with Free Decompression

TL;DR

This paper introduces a novel spectral estimation method called 'free decompression' that leverages free probability theory to infer the eigenvalues of large, impalpable matrices from small submatrix spectral densities, useful in large-scale machine learning.

Contribution

The paper presents a new approach using free probability to estimate eigenvalues of large matrices from limited submatrix data, addressing challenges in distributed and indirect data settings.

Findings

Accurately estimates eigenvalues from small submatrices.

Matches synthetic spectral distributions with theoretical predictions.

Effectively infers spectra of real-world large matrices.

Abstract

Computing eigenvalues of very large matrices is a critical task in many machine learning applications, including the evaluation of log-determinants, the trace of matrix functions, and other important metrics. As datasets continue to grow in scale, the corresponding covariance and kernel matrices become increasingly large, often reaching magnitudes that make their direct formation impractical or impossible. Existing techniques typically rely on matrix-vector products, which can provide efficient approximations, if the matrix spectrum behaves well. However, in settings like distributed learning, or when the matrix is defined only indirectly, access to the full data set can be restricted to only very small sub-matrices of the original matrix. In these cases, the matrix of nominal interest is not even available as an implicit operator, meaning that even matrix-vector products may not be available. In such settings, the matrix is "impalpable," in the sense that we have access to only masked snapshots of it. We draw on principles from free probability theory to introduce a novel method of "free decompression" to estimate the spectrum of such matrices. Our method can be used to extrapolate from the empirical spectral densities of small submatrices to infer the eigenspectrum of extremely large (impalpable) matrices (that we cannot form or even evaluate with full matrix-vector products). We demonstrate the effectiveness of this approach through a series of examples, comparing its performance against known limiting distributions from random matrix theory in synthetic settings, as well as applying it to submatrices of real-world datasets, matching them with their full empirical eigenspectra.