Strictly Low Rank Constraint Optimization -- An Asymptotically $\mathcal{O}(\frac{1}{t^2})$ Method

TL;DR

This paper introduces a novel accelerated proximal gradient method with support set projection for non-convex rank-regularized problems, achieving an optimal convergence rate of O(1/t^2) and promoting strict sparsity in solutions.

Contribution

It proposes a new support set projection technique within proximal gradient descent that guarantees an optimal convergence rate for non-convex rank regularization problems.

Findings

Achieves an O(1/t^2) convergence rate matching Nesterov's optimal rate.

Supports strict sparsity with monotonically shrinking singular value support set.

Introduces a novel support set projection operation for singular values.

Abstract

We study a class of non-convex and non-smooth problems with \textit{rank} regularization to promote sparsity in optimal solution. We propose to apply the proximal gradient descent method to solve the problem and accelerate the process with a novel support set projection operation on the singular values of the intermediate update. We show that our algorithms are able to achieve a convergence rate of $O(\frac{1}{t^2})$, which is exactly same as Nesterov's optimal convergence rate for first-order methods on smooth and convex problems. Strict sparsity can be expected and the support set of singular values during each update is monotonically shrinking, which to our best knowledge, is novel in momentum-based algorithms.