Provably convergent acceleration in factored gradient descent with applications in matrix sensing

TL;DR

This paper proves that accelerated gradient descent applied to non-convex matrix factorization problems converges linearly under certain conditions, with practical acceleration observed in real-world applications like neural activity recovery and quantum tomography.

Contribution

It demonstrates the first provable linear convergence of accelerated gradient descent in non-convex matrix factorization, with insights on parameter selection and practical validation.

Findings

Acceleration achieves linear convergence in non-convex matrix factorization.

Proper selection of acceleration parameters is crucial for convergence.

Practical acceleration observed in neural activity and quantum state applications.

Abstract

We present theoretical results on the convergence of \emph{non-convex} accelerated gradient descent in matrix factorization models with $\ell_2$-norm loss. The purpose of this work is to study the effects of acceleration in non-convex settings, where provable convergence with acceleration should not be considered a \emph{de facto} property. The technique is applied to matrix sensing problems, for the estimation of a rank $r$ optimal solution $X^\star \in \mathbb{R}^{n \times n}$. Our contributions can be summarized as follows. $i)$ We show that acceleration in factored gradient descent converges at a linear rate; this fact is novel for non-convex matrix factorization settings, under common assumptions. $ii)$ Our proof technique requires the acceleration parameter to be carefully selected, based on the properties of the problem, such as the condition number of $X^\star$ and the condition number of objective function. $iii)$ Currently, our proof leads to the same dependence on the condition number(s) in the contraction parameter, similar to recent results on non-accelerated algorithms. $iv)$ Acceleration is observed in practice, both in synthetic examples and in two real applications: neuronal multi-unit activities recovery from single electrode recordings, and quantum state tomography on quantum computing simulators.