$k$-SVD with Gradient Descent

TL;DR

This paper introduces a simple gradient descent approach with a universal step-size rule for computing the top k singular values and vectors of any matrix, achieving global linear convergence and matching the performance of more complex methods.

Contribution

It presents a gradient descent method with a universal step-size for k-SVD that guarantees global convergence for matrices of any rank, improving over prior local or specialized results.

Findings

Gradient descent with a universal step-size converges globally for any k, d ≥ 1.

The method's behavior within an attractive region resembles Heron's method.

Empirical results validate the theoretical convergence guarantees.

Abstract

The emergence of modern compute infrastructure for iterative optimization has led to great interest in developing optimization-based approaches for a scalable computation of $k$-SVD, i.e., the $k\geq 1$ largest singular values and corresponding vectors of a matrix of rank $d \geq 1$. Despite lots of exciting recent works, all prior works fall short in this pursuit. Specifically, the existing results are either for the exact-parameterized (i.e., $k = d$) and over-parameterized (i.e., $k > d$) settings; or only establish local convergence guarantees; or use a step-size that requires problem-instance-specific oracle-provided information. In this work, we complete this pursuit by providing a gradient-descent method with a simple, universal rule for step-size selection (akin to pre-conditioning), that provably finds $k$-SVD for a matrix of any rank $d \geq 1$. We establish that the gradient method with random initialization enjoys global linear convergence for any $k, d \geq 1$. Our convergence analysis reveals that the gradient method has an attractive region, and within this attractive region, the method behaves like Heron's method (a.k.a. the Babylonian method). Our analytic results about the said attractive region imply that the gradient method can be enhanced by means of Nesterov's momentum-based acceleration technique. The resulting improved convergence rates match those of rather complicated methods typically relying on Lanczos iterations or variants thereof. We provide an empirical study to validate the theoretical results.