Efficient Differentiable Approximation of Generalized Low-rank Regularization

TL;DR

This paper introduces an efficient, differentiable approximation for generalized low-rank regularization that enables gradient-based optimization, broadening applicability and improving computational efficiency in machine learning tasks.

Contribution

The authors propose a novel differentiable approximation for generalized low-rank regularization, compatible with gradient-based methods and GPU acceleration, with theoretical convergence guarantees.

Findings

Method is versatile across different LRR forms.

Achieves faster computation compared to traditional SVD-based methods.

Demonstrates effectiveness on various machine learning tasks.

Abstract

Low-rank regularization (LRR) has been widely applied in various machine learning tasks, but the associated optimization is challenging. Directly optimizing the rank function under constraints is NP-hard in general. To overcome this difficulty, various relaxations of the rank function were studied. However, optimization of these relaxed LRRs typically depends on singular value decomposition, which is a time-consuming and nondifferentiable operator that cannot be optimized with gradient-based techniques. To address these challenges, in this paper we propose an efficient differentiable approximation of the generalized LRR. The considered LRR form subsumes many popular choices like the nuclear norm, the Schatten-$p$ norm, and various nonconvex relaxations. Our method enables LRR terms to be appended to loss functions in a plug-and-play fashion, and the GPU-friendly operations enable efficient and convenient implementation. Furthermore, convergence analysis is presented, which rigorously shows that both the bias and the variance of our rank estimator rapidly reduce with increased sample size and iteration steps. In the experimental study, the proposed method is applied to various tasks, which demonstrates its versatility and efficiency. Code is available at https://github.com/naiqili/EDLRR.