Fast Spectrum Estimation of Some Kernel Matrices

TL;DR

This paper introduces a fast eigenvalue quantile estimation method for certain kernel matrices, enabling eigenvalue decay analysis without full matrix construction, useful for low-rank approximation and data dimension studies.

Contribution

The work presents a novel eigenvalue quantile estimation framework for kernel matrices with rapid decay, supported by theoretical bounds and empirical validation, along with a new interlacing theorem.

Findings

Framework provides meaningful eigenvalue bounds without full matrix formation

Empirical results confirm the accuracy of the eigenvalue estimates

Application to intrinsic data dimension analysis demonstrated

Abstract

In data science, individual observations are often assumed to come independently from an underlying probability space. Kernel matrices formed from large sets of such observations arise frequently, for example during classification tasks. It is desirable to know the eigenvalue decay properties of these matrices without explicitly forming them, such as when determining if a low-rank approximation is feasible. In this work, we introduce a new eigenvalue quantile estimation framework for some kernel matrices. This framework gives meaningful bounds for all the eigenvalues of a kernel matrix while avoiding the cost of constructing the full matrix. The kernel matrices under consideration come from a kernel with quick decay away from the diagonal applied to uniformly-distributed sets of points in Euclidean space of any dimension. We prove the efficacy of this framework given certain bounds on the kernel function, and we provide empirical evidence for its accuracy. In the process, we also prove a very general interlacing-type theorem for finite sets of numbers. Additionally, we indicate an application of this framework to the study of the intrinsic dimension of data, as well as several other directions in which to generalize this work.