Stochastic Proximal Gradient Descent for Nuclear Norm Regularization

TL;DR

This paper introduces a stochastic proximal gradient method that significantly reduces space complexity for nuclear norm regularized matrix optimization, achieving efficient convergence rates.

Contribution

It presents a novel stochastic proximal gradient algorithm with $O(m+n)$ space complexity for nuclear norm regularization, improving over previous methods.

Findings

Achieves $O(rac{ ext{log} T}{ ext{sqrt} T})$ convergence for convex functions.

Achieves $O(rac{ ext{log} T}{T})$ convergence for strongly convex functions.

Reduces space complexity from $O(mn)$ to $O(m+n)$.

Abstract

In this paper, we utilize stochastic optimization to reduce the space complexity of convex composite optimization with a nuclear norm regularizer, where the variable is a matrix of size $m \times n$. By constructing a low-rank estimate of the gradient, we propose an iterative algorithm based on stochastic proximal gradient descent (SPGD), and take the last iterate of SPGD as the final solution. The main advantage of the proposed algorithm is that its space complexity is $O(m+n)$, in contrast, most of previous algorithms have a $O(mn)$ space complexity. Theoretical analysis shows that it achieves $O(\log T/\sqrt{T})$ and $O(\log T/T)$ convergence rates for general convex functions and strongly convex functions, respectively.