Efficient and Practical Stochastic Subgradient Descent for Nuclear Norm Regularization

TL;DR

This paper introduces a new stochastic subgradient method for nuclear norm regularized matrix optimization that is computationally efficient, scalable, and maintains low-rank structures, outperforming existing solvers.

Contribution

The authors develop a novel stochastic subgradient approach combining low-rank updates and efficient incremental SVD, enabling practical and scalable solutions for nuclear norm regularized problems.

Findings

Method is computationally cheap and scalable.

Achieves competitive performance with state-of-the-art solvers.

Maintains low-rank factorization for efficient predictions.

Abstract

We describe novel subgradient methods for a broad class of matrix optimization problems involving nuclear norm regularization. Unlike existing approaches, our method executes very cheap iterations by combining low-rank stochastic subgradients with efficient incremental SVD updates, made possible by highly optimized and parallelizable dense linear algebra operations on small matrices. Our practical algorithms always maintain a low-rank factorization of iterates that can be conveniently held in memory and efficiently multiplied to generate predictions in matrix completion settings. Empirical comparisons confirm that our approach is highly competitive with several recently proposed state-of-the-art solvers for such problems.