A Nesterov-style Accelerated Gradient Descent Algorithm for the Symmetric Eigenvalue Problem

TL;DR

This paper introduces a Nesterov-style accelerated gradient descent algorithm on the Grassmann manifold for efficiently computing leading eigenvectors of symmetric matrices, especially effective when the spectral gap is small.

Contribution

It presents a novel accelerated gradient method with constant per-iteration cost and improved complexity over existing methods for the symmetric eigenvalue problem.

Findings

Achieves iteration complexity of  O(1/) with respect to spectral gap .

Outperforms traditional subspace iteration and gradient descent in small spectral gap scenarios.

Matches Lanczos method's complexity with lower per-iteration cost.

Abstract

We develop an accelerated gradient descent algorithm on the Grassmann manifold to compute the subspace spanned by a number of leading eigenvectors of a symmetric positive semi-definite matrix. This has a constant cost per iteration and a provable iteration complexity of $\tilde{\mathcal{O}}(1/\sqrt{\delta})$, where $\delta$ is the spectral gap and $\tilde{\mathcal{O}}$ hides logarithmic factors. This improves over the $\tilde{\mathcal{O}}(1/\delta)$ complexity achieved by subspace iteration and standard gradient descent, in cases that the spectral gap is tiny. It also matches the iteration complexity of the Lanczos method that has however a growing cost per iteration. On the theoretical part, we rely on the formulation of Riemannian accelerated gradient descent by [26] and new characterizations of the geodesic convexity of the symmetric eigenvalue problem by [8]. On the empirical part, we test our algorithm in synthetic and real matrices and compare with other popular methods.