Tractable and Scalable Schatten Quasi-Norm Approximations for Rank Minimization

TL;DR

This paper introduces tractable Schatten quasi-norms, bi-trace and tri-trace, enabling efficient large-scale rank minimization by solving smaller factor matrices, with proven convergence and superior performance.

Contribution

The paper defines new Schatten quasi-norms that are computationally feasible and demonstrates their application to large-scale rank minimization problems with proven convergence.

Findings

Algorithms converge to critical points.

Outperforms state-of-the-art methods in efficiency.

Effective for matrix completion and RPCA.

Abstract

The Schatten quasi-norm was introduced to bridge the gap between the trace norm and rank function. However, existing algorithms are too slow or even impractical for large-scale problems. Motivated by the equivalence relation between the trace norm and its bilinear spectral penalty, we define two tractable Schatten norms, i.e.\ the bi-trace and tri-trace norms, and prove that they are in essence the Schatten-$1/2$ and $1/3$ quasi-norms, respectively. By applying the two defined Schatten quasi-norms to various rank minimization problems such as MC and RPCA, we only need to solve much smaller factor matrices. We design two efficient linearized alternating minimization algorithms to solve our problems and establish that each bounded sequence generated by our algorithms converges to a critical point. We also provide the restricted strong convexity (RSC) based and MC error bounds for our algorithms. Our experimental results verified both the efficiency and effectiveness of our algorithms compared with the state-of-the-art methods.

Tables

Table 1. Table 1: Characteristics of the recommendation datasets.

Dataset	# row	# column	# rating
MovieLens1M	6,040	3,906	1,000,209
MovieLens10M	71,567	10,681	10,000,054
MovieLens20M	138,493	27,278	20,000,263
Netflix	480,189	17,770	100,480,507

Table 2. Table 2: Regularization parameter settings for different algorithms.

Datasets	NNLS	ALT	LRMF	LMaFit	IRLS	Ours
	$\mu$	$\lambda$	$\lambda$	$\lambda$	$\lambda$	$\mu$
MovieLens1M	1.70	50	5	–	1e-6	100
MovieLens10M	4.80	100	5	–	1e-6	100
MovieLens20M	6.63	150	5	–	1e-6	100
Netflix	16.76	150	5	–	1e-6	100

Equations

$$\Vert X{\Vert }_{S_p}={\left({\sum }_{i=1}^n{\sigma }_i^p(X)\right)}^{1/p},$$

$$\underset{X\in {\mathbb{R}}^{m\times n}}{\min }\left\{\Vert X{\Vert }_{S_p}^p:\mathcal{A}(X)=b\right\},$$

$$\underset{X\in {\mathbb{R}}^{m\times n}}{\min }\left\{\Vert X{\Vert }_{S_p}^p+\frac{1}{\mu }f\left(\mathcal{A}(X)-b\right)\right\},$$

$$\begin{array}{rl}\Vert X{\Vert }_{\text{tr}}& =\underset{U\in {\mathbb{R}}^{m\times d},V\in {\mathbb{R}}^{n\times d}:X=U{V}^T}{\min }\Vert U{\Vert }_F\Vert V{\Vert }_F\\ & =\underset{U,V:X=U{V}^T}{\min }\frac{\Vert U{\Vert }_F^2+\Vert V{\Vert }_F^2}{2}.\end{array}$$

$$\Vert X{\Vert }_{\text{Bi-tr}}:=\underset{U,V:X=U{V}^T}{\min }\Vert U{\Vert }_{\text{tr}}\Vert V{\Vert }_{\text{tr}}.$$

$$\Vert X{\Vert }_{\text{Bi-tr}}=\Vert X{\Vert }_{S_{1/2}},$$

$$\begin{array}{rl}\Vert X{\Vert }_{\text{Bi-tr}}& =\underset{U,V:X=U{V}^T}{\min }\Vert U{\Vert }_{\text{tr}}\Vert V{\Vert }_{\text{tr}}=\underset{U,V:X=U{V}^T}{\min }\frac{\Vert U{\Vert }_{\text{tr}}^2+\Vert V{\Vert }_{\text{tr}}^2}{2}\\ & =\underset{U,V:X=U{V}^T}{\min }{\left(\frac{\Vert U{\Vert }_{\text{tr}}+\Vert V{\Vert }_{\text{tr}}}{2}\right)}^2.\end{array}$$

$$\Vert X{\Vert }_{\text{Tri-tr}}:=\underset{U,V,W:X=UV{W}^T}{\min }\Vert U{\Vert }_{\text{tr}}\Vert V{\Vert }_{\text{tr}}\Vert W{\Vert }_{\text{tr}}.$$

$$\Vert X{\Vert }_{\text{Tri-tr}}=\Vert X{\Vert }_{S_{1/3}}.$$

$$\begin{array}{rl}& \Vert X{\Vert }_{\text{Tri-tr}}=\underset{X=UV{W}^T}{\min }{\left(\frac{\Vert U{\Vert }_{\text{tr}}+\Vert V{\Vert }_{\text{tr}}+\Vert W{\Vert }_{\text{tr}}}{3}\right)}^3\\ =& \underset{X=UV{W}^T}{\min }\Vert U{\Vert }_{\text{tr}}\Vert V{\Vert }_{\text{tr}}\Vert W{\Vert }_{\text{tr}}=\underset{X=UV{W}^T}{\min }\frac{\Vert U{\Vert }_{\text{tr}}^3+\Vert V{\Vert }_{\text{tr}}^3+\Vert W{\Vert }_{\text{tr}}^3}{3}.\end{array}$$

$$\begin{array}{rl}\Vert X{\Vert }_{\text{tr}}\le \Vert X& {\Vert }_{\text{Bi-tr}}\le r\Vert X{\Vert }_{\text{tr}},\\ \Vert X{\Vert }_{\text{tr}}\le \Vert X{\Vert }_{\text{Bi-tr}}& \le \Vert X{\Vert }_{\text{Tri-tr}}\le r^2\Vert X{\Vert }_{\text{tr}}.\end{array}$$

$$\Vert x{\Vert }_1\le \Vert x{\Vert }_{p_1},\Vert x{\Vert }_{p_1}\le \Vert x{\Vert }_{p_2}\le n^{1/p_2-1/p_1}\Vert x{\Vert }_{p_1}.$$

$$\begin{array}{rl}& \Vert X{\Vert }_{\text{tr}}=\Vert \text{diag}(\Sigma ){\Vert }_1\le \Vert \text{diag}(\Sigma ){\Vert }_{\frac{1}{2}}=\Vert X{\Vert }_{\text{Bi-tr}}\le r\Vert X{\Vert }_{\text{tr}},\\ & \Vert X{\Vert }_{\text{tr}}=\Vert \text{diag}(\Sigma ){\Vert }_1\le \Vert \text{diag}(\Sigma ){\Vert }_{\frac{1}{3}}=\Vert X{\Vert }_{\text{Tri-tr}}\le r^2\Vert X{\Vert }_{\text{tr}}.\end{array}$$

$$\begin{array}{rl}& \Vert X{\Vert }_{\text{tr}}=\Vert \text{diag}(\Sigma ){\Vert }_1\le \Vert \text{diag}(\Sigma ){\Vert }_{\frac{1}{2}}\\ & \end{array}$$