Spectral Smoothing via Random Matrix Perturbations

TL;DR

This paper introduces a stochastic spectral smoothing technique using random matrix perturbations, providing new bounds for the maximum eigenvalue function and applying these results to improve online learning algorithms for variance minimization and PCA.

Contribution

It develops a novel spectral smoothing method based on random matrix theory and derives state-of-the-art bounds for eigenvalue functions, enabling improved online optimization algorithms.

Findings

Achieves new smoothing bounds for maximum eigenvalue using GOE.

Provides an online variance minimization algorithm with $O((N ext{log} N)^{1/4} ext{sqrt} T)$ regret.

Extends the approach to online PCA with similar regret guarantees.

Abstract

We consider stochastic smoothing of spectral functions of matrices using perturbations commonly studied in random matrix theory. We show that a spectral function remains spectral when smoothed using a unitarily invariant perturbation distribution. We then derive state-of-the-art smoothing bounds for the maximum eigenvalue function using the Gaussian Orthogonal Ensemble (GOE). Smoothing the maximum eigenvalue function is important for applications in semidefinite optimization and online learning. As a direct consequence of our GOE smoothing results, we obtain an $O((N \log N)^{1/4} \sqrt{T})$ expected regret bound for the online variance minimization problem using an algorithm that performs only a single maximum eigenvector computation per time step. Here $T$ is the number of rounds and $N$ is the matrix dimension. Our algorithm and its analysis also extend to the more general online PCA problem where the learner has to output a rank $k$ subspace. The algorithm just requires computing $k$ maximum eigenvectors per step and enjoys an $O(k (N \log N)^{1/4} \sqrt{T})$ expected regret bound.