Convergence Analysis of the Rank-Restricted Soft SVD Algorithm

TL;DR

This paper analyzes the convergence of the Rank-Restricted Soft SVD algorithm, identifies issues with standard convergence, and proposes modifications ensuring linear convergence for large sparse matrix decompositions.

Contribution

It provides a convergence analysis of RRSS, introduces a sign-consistency modification, and proves linear convergence of singular vectors and values.

Findings

Standard RRSS may not converge without modification

Sign consistency in singular vectors ensures convergence

Linear convergence to soft thresholded singular values

Abstract

The soft SVD is a robust matrix decomposition algorithm and a key component of matrix completion methods. However, computing the soft SVD for large sparse matrices is often impractical using conventional numerical methods for the SVD due to large memory requirements. The Rank-Restricted Soft SVD (RRSS) algorithm introduced by Hastie et al. addressed this issue by sequentially computing low-rank SVDs that easily fit in memory. We analyze the convergence of the standard RRSS algorithm and we give examples where the standard algorithm does not converge. We show that convergence requires a modification of the standard algorithm, and is related to non-uniqueness of the SVD. Our modification specifies a consistent choice of sign for the left singular vectors of the low-rank SVDs in the iteration. Under these conditions, we prove linear convergence of the singular vectors using a technique motivated by alternating subspace iteration. We then derive a fixed point iteration for the evolution of the singular values and show linear convergence to the soft thresholded singular values of the original matrix. This last step requires a perturbation result for fixed point iterations which may be of independent interest.