Projection Free Rank-Drop Steps

TL;DR

This paper introduces a rank-drop method integrated with the Frank-Wolfe algorithm to maintain low-rank solutions in nuclear norm constrained problems, reducing computational costs.

Contribution

It proposes a novel rank-drop step that decreases the rank during optimization, enhancing efficiency in projection-free algorithms like Frank-Wolfe.

Findings

Rank-drop steps significantly reduce the solution rank.

The method maintains feasibility and can be integrated into existing algorithms.

Experimental results show improved efficiency over standard Frank-Wolfe.

Abstract

The Frank-Wolfe (FW) algorithm has been widely used in solving nuclear norm constrained problems, since it does not require projections. However, FW often yields high rank intermediate iterates, which can be very expensive in time and space costs for large problems. To address this issue, we propose a rank-drop method for nuclear norm constrained problems. The goal is to generate descent steps that lead to rank decreases, maintaining low-rank solutions throughout the algorithm. Moreover, the optimization problems are constrained to ensure that the rank-drop step is also feasible and can be readily incorporated into a projection-free minimization method, e.g., Frank-Wolfe. We demonstrate that by incorporating rank-drop steps into the Frank-Wolfe algorithm, the rank of the solution is greatly reduced compared to the original Frank-Wolfe or its common variants.

Tables

Table 1. Table 1 : Computational results on matrix completion problems averaged over 5 random initializations. The max rank is the maximum rank observed over all 5 trials. For ML-20M, the FW and AFW algorithms took too long to successfully terminate.

Dataset		Frank-Wolfe	Away-Step FW	In-Face$(0,\infty )$	ActiveALT	Rank-Drop FW
ML-100k	RMSE	0.878	0.878	0.878	0.877	0.879
	Rank (Max)	501.4 (504)	466.8 (477)	54.2 (64)	43.4 (50)	41.6 (44)
	Time (s)	146.09	152.49	37.85	30.28	35.45
ML-1M	RMSE	0.820	0.820	0.822	0.818	0.820
	Rank (Max)	467.4 (478)	425.2 (430)	77.6 (112)	65 (68)	67 (69)
	Time (s)	1,127.70	1,197.67	356.42	503.39	284.85
ML-10M	RMSE	0.806	0.806	0.807	0.806	0.808
	Rank (Max)	1000 (1000)	889.8 (893)	210 (297)	141.8 (144)	140 (140)
	Time (s)	41,467.60	44,327.57	29,630.43	13,281.95	12,499.43
ML-20M	RMSE	-	-	0.800	0.800	0.801
	Rank (Max)	-	-	274.6 (471)	206.2 (214)	202 (203)
	Time (s)	-	-	117,535.82	38,497.16	29,102.62

Table 2. Table 2 : MovieLens Data

Dataset	# Users	# Movies	# Ratings
MovieLens 100k	943	1,682	100,000
MovieLens 1M	6,040	3,900	1,000,209
MovieLens 10M	82,248	10,681	10,000,054
MovieLens 20M	138,493	27,278	20,000,263

Table 3. Table 3 : Parameters used for each dataset.

Dataset	Frank-Wolfe $\delta$	ActiveALT $\lambda$
MovieLens 100k	3	10.94
MovieLens 1M	3.4	22.7
MovieLens 10M	5.2	49.4
MovieLens 20M	6.6	59.04

Equations

$$\underset{X\in {\mathbb{R}}^{m\times n}}{\min }f(X)s.t.\Vert X{\Vert }_{NN}\le \delta$$

$$\underset{x\in \mathcal{S}}{\min }f(x)$$

$$v_k:=\underset{v\in {\mathcal{A}}_k}{\mathrm{argmin}}⟨\nabla f(x_k),x_k-v_k⟩$$

$$\mathcal{F}(X_k)=\{\begin{array}{l}{\mathcal{B}}_{NN}(0,\delta )\text{, when ∥Xk∥<δ}\\ UM{V}^{\top },\text{ otherwise, }\end{array}$$

$$Z_k:=\underset{Z\in \mathcal{F}(X_k)}{\mathrm{argmax}}⟨\nabla f(X_k),Z⟩.$$

$$\begin{array}{rl}\mathcal{R}(A):=& \{{\sigma }^{-1}u{v}^{\top }:\exists x\in {\mathbb{R}}^n,y\in {\mathbb{R}}^m\\ & s.t.u=Ax,v=A^{\top }y,\sigma =y^{\top }Ax\ne 0\}.\end{array}$$

$$\mathcal{R}(A)=\left\{\frac{Us{t}^{\top }{V}^{\top }}{s^{\top }{\Sigma }^{-1}t},s^{\top }{\Sigma }^{-1}t>0\right\}.$$

$$X_{k+1}:=X_k+\tau D_{rd},\text{ where }{D}_{rd}:=X_k-\delta {\hat{Z}}_k.$$

$$\kappa (X_k):=\frac{\delta -\Vert X_k{\Vert }_{NN}}{2}$$

$$\begin{array}{rl}\underset{Z}{\min }& ⟨-Z,\nabla f(X_k)⟩\\ s.t.& Z\in {\mathcal{B}}_{NN}(0,\kappa (X_k))\cap \mathcal{R}(X_k).\end{array}$$

$$\begin{array}{rl}\underset{s,t\in {\mathbb{R}}^r}{\min }& ⟨X_k-\frac{Us{t}^{\top }{V}^{\top }}{s^{\top }{\Sigma }^{-1}t},\nabla f(X_k)⟩\\ s.t.& \frac{Us{t}^{\top }{V}^{\top }}{s^{\top }{\Sigma }^{-1}t}\in {\mathcal{B}}_{NN}(0,\kappa (X_k))\\ & s^{\top }{\Sigma }^{-1}t>0.\end{array}$$

$$\begin{array}{rl}\underset{s,t\in {\mathbb{R}}^r}{\min }& q(s,t):=⟨\nabla f(X_k),-\kappa (X_k)Us{t}^{\top }{V}^{\top }⟩\\ s.t.& s^{\top }{\Sigma }^{-1}t=\kappa (X_k{)}^{-1}\\ & \Vert s{\Vert }_2=1,\Vert t{\Vert }_2\le 1\end{array}$$

$$\left(\underset{G\in \partial \Vert X_k{\Vert }_{NN}}{\max }⟨D_k,G⟩\right)\le 0,$$

$$\left(\underset{G\in \partial \Vert X_k{\Vert }_{NN}}{\max }⟨D_k,G⟩\right)\le 0$$

$$\begin{array}{r}\underset{s\in {\mathbb{R}}^r}{\max }\frac{1}{2}\frac{s^{\top }(W^{\top }+W)s}{s^{\top }{\Sigma }^{-1}s}s.t.\Vert s{\Vert }_2=1.\end{array}$$

$$\mathcal{L}(s,t,\lambda ,\alpha ,\beta ):=$$

$$+\alpha (s^{\top }s-1)+\beta (t^{\top }t-1).$$

$$Vt=A^{\top }y=V\Sigma U^{\top }y\Rightarrow {\Sigma }^{-1}t$$

$$\sigma =y^{\top }Ax=y^{\top }(Us)=(U^{\top }y{)}^{\top }s=t^{\top }{\Sigma }^{-1}s.$$

$$\begin{array}{rl}{\sigma }^{-1}\Vert u{\Vert }_2\Vert v{\Vert }_2=\frac{\Vert Us{\Vert }_2\Vert Vt{\Vert }_2}{|s^{\top }{\Sigma }^{-1}t|}& \ge \frac{\Vert s{\Vert }_2\Vert t{\Vert }_2}{\Vert s{\Vert }_2\Vert {\Sigma }^{-1}t{\Vert }_2}\\ & \ge \frac{1}{{\max }_{z:\Vert z{\Vert }_2=1}\Vert {\Sigma }^{-1}z{\Vert }_2}\\ & ={\sigma }_r(A)\end{array}$$

$$\Vert R_k{\Vert }_{NN}+\Vert Z_k{\Vert }_{NN}$$

$$\le \Vert X{\Vert }_k+2\Vert Z_k{\Vert }_{NN}$$

$$\le \delta$$

$$X_k={\alpha }_0\delta {\hat{Z}}_k+\sum _{i=1}^{r-1}{\alpha }_i\delta u_i{v}_i^{\top }$$

$$\begin{array}{rl}\Vert R_k{\Vert }_{NN}+\Vert Z_k{\Vert }_{NN}& \le \delta \\ \Rightarrow \delta \left(\sum _{i=1}^{r-1}{\alpha }_i\right)+\delta {\alpha }_0& \le \delta \\ \Rightarrow \sum _{i=0}^{r-1}{\alpha }_i& \le 1\end{array}$$

$$X_{k+1}$$

$$\displaystyle=(1+\tau)R_{k}+(1+\tau)\alpha_{0}\delta\hat{Z}_{k}-\tau\delta\hat{Z}_{k}~{}~{}\text{(using \eqref{eq:cvx_combo})}$$

$$=(1+\tau )R_k+({\alpha }_0+{\alpha }_0\tau -\tau )\delta {\hat{Z}}_k$$

$$=\sum _{i=1}^{r-1}(1+\tau ){\alpha }_i\delta u_i{v}_i^{\top }+({\alpha }_0+{\alpha }_0\tau -\tau )\delta {\hat{Z}}_k$$

$$=\sum _{i=1}^{r-1}{\beta }_i\delta u_i{v}_i^{\top }+{\beta }_0\delta {\hat{Z}}_k$$

$$\sum _{i=0}^{r-1}{\beta }_i$$

$$=(1+\tau )\sum _{i=0}^{r-1}{\alpha }_i-\tau$$

$$\le (1+\tau )-\tau$$

$$=1.$$

$$\begin{array}{rl}\Vert Z{\Vert }_{NN}& ={\Vert \frac{Us{t}^{\top }{V}^{\top }}{s^{\top }{\Sigma }^{-1}t}\Vert }_{NN}\\ & \le \Vert Us{\Vert }_2\Vert Vt{\Vert }_2(\Vert {\Sigma }^{-1}s{\Vert }_2\Vert t{\Vert }_2{)}^{-1}\\ & \le {\sigma }_r(X_k)\\ & \le \kappa (X_k)\end{array}$$

Full text

1 Introduction

The Frank-Wolfe algorithm has been widely used for many machine learning applications due to its projection-free property, particularly when linear minimization on the domain is easy but projection is difficult. A particularly interesting problem in machine learning is the nuclear norm constrained problem,

[TABLE]

where $\lVert\cdot\rVert_{NN}$ is the nuclear norm, which is a typical convex relaxation for rank constrained optimization problems Fazel et al. (2001). Common applications of nuclear-norm constrained problems are matrix completions, multivariate regression, multi-task learning, and clustering with missing information. For (1), a projection operation onto the nuclear norm ball will require a full singular value decomposition (SVD), which can be too expensive to perform at each iteration, but is required by methods such as the projected gradient descent.

Without any projection, the FW iterate is guaranteed to be feasible. For (1), the linear subproblem used by FW only requires computing the singular vector pair corresponding to the largest singular value of the gradient at each iteration Jaggi (2013). This is significantly cheaper than computing the full SVD when the dimension of the matrix is large. A challenge when using FW for (1) is that the nuclear norm ball is a convex hull of an infinite number of rank-one matrices, referred to as the atoms of the set of feasible points. Since an atom is added at each iteration, the solution is expressed as a convex combination of an arbitrarily large atomic set, which lacks the crucial low-rank property we desire from nuclear norm constrained problems.

While other methods exist for solving instances of (1), e.g., Active Subspace Selection Hsieh and Olsen (2014) or Redistributing Nonconvex Regularizers Yao et al. (2016), implementations for these methods utilize specific properties of $f(\cdot)$. In Yao et al. (2016), it appears that for each choice of $f(\cdot)$, a specialized solver must be written, and in Hsieh and Olsen (2014), knowledge of special structures using $\nabla f(\cdot)$ and $\nabla^{2}f(\cdot)$ is necessary to use the approach efficiently. In contrast, the FW can be readily used for a general $f(\cdot)$ without special modifications.

One of our main goals is to devise a new way to efficiently generate descent feasible steps which also decrease the rank of the current iterate when solving (1) without assuming a specific form for $f(\cdot)$. We propose a new nonconvex optimization formulation to determine the steepest descent rank-drop steps. By further considering the interior rank-drop step and exterior rank-drop step cases separately, we obtain formulations which lead to efficient feasible rank-drop step computation.

We establish theoretical properties of the proposed formulations. In addition we demonstrate computationally that the proposed rank-drop FW method can obtain much lower rank solutions than previous FW alternatives, greatly improving computational efficiency.

1.1 Notation

In this paper, $\sigma_{i}(A)$ denotes the $i^{\textsuperscript{th}}$ singular value of $A$, with $\sigma_{1}\geq...\geq\sigma_{\min\{m,n\}}$. For a matrix $A=[a_{ij}]\in\mathbb{R}^{m\times n}$, $\lVert A\rVert_{NN}\coloneqq\sum_{i}\sigma_{i}(A)$ denotes the nuclear norm and $\lVert A\rVert_{sp}\coloneqq\sigma_{1}(A)$ denotes the spectral norm. Let $\mathcal{B}_{\lVert\cdot\rVert}(A,\epsilon)\coloneqq\{X\in\mathbb{R}^{m\times n}:\lVert A-X\rVert\leq\epsilon\}$ be the closed norm-ball with a specified norm. We use the term thin SVD to mean the SVD with strictly positive singular values, i.e., if $A=U\Sigma V^{\top}$ is a thin SVD and $rank(A)=r$, then $U\in\mathbb{R}^{m\times r},\Sigma\in\mathbb{R}^{r\times r}$, and $V\in\mathbb{R}^{n\times r}$. Finally, $\langle A,B\rangle\coloneqq\text{tr}(A^{\top}B)$ denotes the usual trace inner product.

2 The Frank-Wolfe Algorithm and Away Step

We briefly review FW along with its common variants. Here FW is described for a general problem,

[TABLE]

where $\mathcal{S}$ is a closed and bounded convex set.

The FW algorithm can be summarized in Algorithm 1. We denote the FW direction as $d_{fw}\coloneqq s_{k}-x_{k}$.

There are many choices for the stepsize $\tau$ which ensure convergence. We will assume in this paper that the optimal step-size is used since the stepsize choice is not the focus of this work.

Away steps has been introduced in Wolfe (1970) and analyzed in Guélat and Marcotte (1986) to move away from “bad atoms” to accelerate convergence. The away step method maintains a set of active atoms, $\mathcal{A}_{k}$, such that the current iterate $x_{k}$ is decomposed into an atomic decomposition, i.e., $x_{k}=\sum_{a_{i}\in\mathcal{A}_{k}}\alpha_{a_{i}}a_{i}$, where the weight satisfy $\alpha_{a_{i}}>0$ and $\sum_{a_{i}\in\mathcal{A}_{k}}\alpha_{a_{i}}\leq 1$.

The away step provides a direction that moves away from one of the active atoms which yields a better descent direction than the current FW direction. This direction solves the following optimization problem,

[TABLE]

To ensure convergence, the away direction $d_{away}\coloneqq x_{k}-v_{k}$ is only taken if $\langle\nabla f(x_{k}),d_{away}\rangle\leq\langle\nabla f(x_{k}),d_{fw}\rangle$. Algorithm 2 summarizes the away step FW method (AFW) in Lacoste-Julien and Jaggi (2015).

2.1 In-Face Steps

In Freund et al. (2015), In-Face Steps are proposed to generalize away-steps. Instead of generating descent directions by moving away from bad atoms, the in-face step selects the best descent direction along the minimal face containing the current iterate. By aiming to move along the minimal face to the boundary of the feasible region to reach a lower dimensional face, the algorithm explicitly maintains a low-rank structure without compromising convergence. For (1), as in Freund et al. (2015), the minimal face $\mathcal{F}(X_{k})$ of $\mathcal{B}_{NN}(0,\delta)$ containing a point $X_{k}$ is given by the set,

[TABLE]

where $X_{k}$ has a thin $r$ rank SVD, $U\Sigma V^{\top}$, $M$ is a real positive semidefinite matrix with $\text{tr}(M)=\delta$.

In particular, for comparison, we consider the In-Face step from the Away-step Strategy described in Freund et al. (2015), which uses the following direction,

[TABLE]

3 Rank-Drop Steps

One significant issue with the FW method for nuclear norm constrained problems is that the rank of the intermediate solution often steadily increases Freund et al. (2015), and frequently yielding a high rank solution upon termination. In Figure 1, we highlight this phenomenon. For very large problems, we cannot store the matrix $X_{k}$, and instead, only maintain the rank-one factors. Thus, computing the gradient at each iteration can be computationally prohibitive. For matrix completion, as an example, if $\Omega$ is the set of observed entries, then forming the gradient requires at least $\lvert\Omega\rvert\cdot r$ inner product calculations. Since the number of observed entries, $\lvert\Omega\rvert$, is typically very large, any increase in the rank of the solution greatly increases the computational time in each iteration.

To address this issue, we propose to search over the set of rank-one matrices which decrease the rank of the iterate by one. The proposed optimization formulation for determining such a rank-one matrix is motivated by the following theorem.

Theorem 3.1** (Egerváry (1960)).**

Let $u\in\mathbb{R}^{m}$, $v\in\mathbb{R}^{n}$, $A\in\mathbb{R}^{m\times n}$, and $B=A-\sigma^{-1}uv^{\top}$. Then $rank(B)=rank(A)-1$ if and only if there are vectors $x\in\mathbb{R}^{n}$ and $y\in\mathbb{R}^{m}$ such that $u=Ax$, $v=A^{\top}y$ and $\sigma=y^{\top}Ax\neq 0$.

3.1 The Rank-Drop Optimization Framework

Assume that $rank(X_{k})>1$. Our goal is to determine a rank-drop step which reduces the rank of $X_{k}$ but has a comparable objective value. We restrict our attention to the rank-one matrices that can decrease the rank of a matrix; let ${\mathcal{R}}(A)$ denote, for a matrix $A$, the set of rank-drop steps, which are these rank-one matrices,

[TABLE]

Since $u$ and $v$ must be in the column and row spans of $A$ respectively, Lemma 1 shows that the set of rank-drop steps can be expressed in a more concise form.

Lemma 3.2**.**

111Due to space limitations, proofs will be omitted. The proofs will be added in an appendix at https://arxiv.org/abs/1704.04285.

Let $A\in\mathbb{R}^{m\times n}$ with a thin SVD $A=U\Sigma V^{\top}$. Then we can rewrite the set of rank-drop steps as,

[TABLE]

Let $Z_{k}\in\mathcal{R}(X_{k})$ and $\hat{Z}_{k}=Z_{k}/\lVert Z_{k}\rVert_{NN}$. Then, similar to away steps, we consider iterates in the following form,

[TABLE]

Our goal is to find a rank-drop step that results in a feasible iterate which best minimizes the objective. Since not all rank-drop steps lead to feasible iterates and testing for feasibility at each iteration is computationally expensive, we want to establish a verifiable sufficient condition for rank-drop steps that ensure feasibility. In Lemma 3.3, we first establish a lower bound on the nuclear norm of the rank-drop step.

Lemma 3.3**.**

Let $A\in\mathbb{R}^{m\times n}$ with rank $r$ and let $Z\in\mathcal{R}(A)$ be an arbitrary rank-drop step. Then $\lVert Z\rVert_{NN}\geq\sigma_{r}(A)$.

To ensure feasibility of the iterate after a rank-drop step, we consider two distinct cases. Let $\kappa(X_{k})$ be half of the distance between $X_{k}$ and the boundary of the nuclear norm ball,

[TABLE]

We consider $\kappa(X_{k})\geq\sigma_{r}(X_{k})$, the interior case, and $\kappa(X_{k})<\sigma_{r}(X_{k})$, the exterior case. For each case, we will motivate a rank-drop step formulation to guarantee feasibility of the next iterate and provide a method for its computation.

3.2 The Interior Rank-Drop Problem

Assume that $\kappa(X_{k})\geq\sigma_{r}(X_{k})$, where $\kappa(X_{k})$ is defined in (9). Theorem 3.4 below establishes an easily verifiable condition guaranteeing that $X_{k+1}$, after taking the rank-drop step, remains in the interior of the nuclear ball.

Theorem 3.4**.**

Let $X_{k}\in\mathbb{R}^{m\times n}$ with $\lVert X_{k}\rVert_{NN}<\delta$, $Z_{k}\in\mathcal{R}(A)$, and $\hat{Z}_{k}=Z_{k}/\lVert Z_{k}\rVert_{NN}$. If $\lVert Z_{k}\rVert\leq(\delta-\lVert X_{k}\rVert_{NN})/2$, then $X_{k+1}=X_{k}+\tau(X_{k}-\delta\hat{Z}_{k})\in\mathcal{B}_{NN}(0,\delta)$ for every $\tau\in[0,\tau^{*}]$, where $\tau^{*}=\lVert Z_{k}\rVert_{NN}/(\delta-\lVert Z_{k}\rVert_{NN})$. Moreover, if $\tau=\tau^{*}$, then $rank(X_{k+1})=rank(X_{k})-1$.

Theorem 3.4 motivates the following formulation,

[TABLE]

Problem (10) determines, in the interior case, the best rank-drop step for solving (1) based on the first order approximation to the objective. The constraint on $Z$ guarantees that the next iterate is feasible if a rank-drop step is taken.

Theorem 3.5**.**

If $rank(X_{k})\geq 1$ with $\sigma_{r}(X_{k})\leq\kappa(X_{k})$, then the feasible region for (10) is non-empty.

Assume that a thin SVD for $X_{k}$, $X_{k}=U\Sigma V^{\top}$, is given. Using Lemma 1, the constraint in (10) can be made explicit,

[TABLE]

To make (11) more amenable to computation, we remove the fraction using the normalization constraint $s^{\top}\Sigma^{-1}t=\kappa(X_{k})^{-1}$ and formulate the problem equivalently as follows,

[TABLE]

Note that the constraints in (12) also ensure that $s$ and $t$ cannot be rescaled to obtain a different solution yielding an identical rank-drop step. The equivalence of (10) and (12) is formally established in Theorem 3.6.

Theorem 3.6**.**

If $X_{k}$ is a feasible point of (1) and $\kappa(X_{k})\geq\sigma_{r}(X_{k})$, then an optimal solution to (12) is an optimal solution to (11). Moreover, an optimal to (11) can always be rescaled into an optimal solution to (12).

3.3 The Exterior Rank-Drop Problem

Now consider the case $\kappa(X_{k})<\sigma_{r}(X_{k})$. If $\kappa(X_{k})$ is relatively large and $X_{k}$ is not nearly rank deficient, rank-drop is less important. If $\kappa(X_{k})$ is small, however, then $X_{k}$ is close to the boundary of the nuclear ball. In this case, we are interested in rank-drop descent directions that move inside the nuclear norm ball. We establish the following theorems to facilitate formulating appropriate optimization problems for this case. Subsequently $\partial g(X)$ denotes the subdifferential of $g(X)$ at $X$.

Theorem 3.7**.**

Let $X_{k}\in\mathcal{B}_{NN}(0,\delta)$ have the thin SVD $X_{k}=U\Sigma V^{\top}$. Define $D_{k}=X_{k}-\delta\hat{Z}_{k}$ with $Z_{k}\in\mathcal{R}(X_{k})$ and $\hat{Z}_{k}=Z_{k}/\lVert Z_{k}\rVert_{NN}=Ust^{\top}V^{\top}$, for some $s$ and $t$ with $\lVert s\rVert=\lVert t\rVert=1$. Then,

[TABLE]

if and only if $\delta s^{\top}t\geq\lVert X_{k}\rVert_{NN}$.

Following Theorem 3.7, $D_{k}$ will be a descent direction for the nuclear norm at $X_{k}$ if $\delta s^{\top}t\geq\lVert X_{k}\rVert_{NN}$.

When $X_{k}$ is on the boundary of the nuclear norm ball, Corollary 3.7.1 below states a simpler and useful characterization.

Corollary 3.7.1**.**

Let $X_{k}$ have the thin SVD $X_{k}=U\Sigma V^{\top}$ with $\lVert X_{k}\rVert_{NN}=\delta$. Define $D_{k}=X_{k}-\delta\hat{Z}_{k}$ with $Z_{k}\in\mathcal{R}(X_{k})$ and $\hat{Z}_{k}=Z_{k}/\lVert Z_{k}\rVert_{NN}=Ust^{\top}V^{\top}$, for some $s$ and $t$ with $\lVert s\rVert_{2}=\lVert t\rVert_{2}=1$. Then,

[TABLE]

if and only if $s=t$.

When $X_{k}$ is on the boundary of the nuclear ball, $\kappa(X_{k})=0$. Lemma 3.8 below shows that $X_{k}$ is close to the boundary when either $r$ is large or when $\sigma_{r}(X_{k})$ is small, assuming $\kappa(X_{k})<\sigma_{r}(X_{k})$.

Lemma 3.8**.**

If $\kappa(X_{k})<\sigma_{r}(X_{k})$, then $\lVert X_{k}\rVert_{NN}>\frac{r}{r+2}\delta$.

For the exterior case, $\kappa(X_{k})<\sigma_{r}(X_{k})$, we are mostly interested in the situation when $X_{k}$ is close to the boundary of the nuclear norm ball. Consequently, we motivate the next formulation assuming $\lVert X_{k}\rVert_{NN}=\delta$ as a good approximation.

When $X_{k}$ is on the boundary of the nuclear norm ball, we consider all directions which move into the nuclear norm ball, hence feasible directions, as candidate solutions. From Corollary 3.7.1, this can be reduced to searching over directions $D_{k}=X_{k}-\delta Uss^{\top}V^{\top}$. Let $W=U^{\top}\nabla f(X_{k})V$. Note that $\frac{1}{2}s^{\top}(W^{\top}+W)s=\langle\nabla f(X_{k}),Uss^{\top}V^{\top}\rangle$. Hence, in the exterior case, we solve the following optimization problem,

[TABLE]

We note that (15) is a generalized eigenvalue problem, which can be further reduced to a standard eigenvalue problem since $\Sigma^{-1}$ is a nonsingular diagonal matrix.

Theorem 3.9 below establishes that, using (15), the required step-size to decrease the rank of the current iterate also guarantees that the next iterate is feasible.

Theorem 3.9**.**

Let $s$ be an optimal solution to $\eqref{ann_rank_drop}$ and let $D_{k}=X_{k}-\delta Uss^{\top}V^{\top}$, where $X_{k}=U\Sigma V^{\top}$ is a thin SVD with $\lVert X_{k}\rVert_{NN}\leq\delta$. If $\tau^{*}=(\delta s^{\top}\Sigma^{-1}s-1)^{-1}$ and $X_{k+1}=X_{k}+\tau^{*}D_{k}$, then $rank(X_{k+1})=rank(X_{k})-1$ and $\lVert X_{k+1}\rVert_{NN}\leq\delta$.

We observe that $\tau^{*}$ matches the upper bound on the step-size for in-face steps when an away-step method is used and when $\lVert X_{k}\rVert=\delta$.

3.3.1 Comparison with In-Face Steps

Lemma 3.10 and 3.11 below show that the rank-drop steps derived from (12) and (15) lie on the minimal face of $\mathcal{B}_{NN}(0,\delta)$ containing $X_{k}$, denoted as $\mathcal{F}(X_{k})$.

Lemma 3.10**.**

Suppose $\lVert X_{k}\rVert_{NN}<\delta$ and $\kappa(X_{k})\geq\sigma_{r}(X_{k})$, with thin SVD $X_{k}=U\Sigma V^{\top}$. Then $Z=\kappa(X_{k})Ust^{\top}V^{\top}\in\mathcal{F}(X_{k})$, where $(s,t)$ is the solution to (12).

Lemma 3.11**.**

Suppose $\kappa(X_{k})<\sigma_{r}(X_{k})$, with thin SVD $X_{k}=U\Sigma V^{\top}$. Then $Z=\delta Uss^{\top}V^{\top}\in\mathcal{F}(X_{k})$, where $s$ is the solution to (15).

We emphasize, however, that the rank-drop steps are constructed explicitly to decrease the rank of the current iterate. Specifically, we highlight the following differences between rank-drop steps and the in-face direction (5) suggested in Freund et al. (2015). Firstly, when the iterate is in the interior of the nuclear norm ball, the in-face steps often do not lead to rank-drop steps. Moreover, at each iteration when the iterate is in the interior, a binary search is required to determine the maximum feasible step length once the direction is computed. This requires several SVD updates, leading to very expensive intermediate iterates. Thus, it is critical for the iterates of the in-face method to reach the boundary of the nuclear norm ball quickly. Secondly, the parametrization suggested in Freund et al. (2015) (i.e. $\gamma_{1}=0$ and $\gamma_{2}=\infty$) for large datasets corresponds to only taking the maximum step length. On the boundary of the nuclear norm ball, this is equivalent to only accepting iterates with rank decreased. Since our proposed rank-drop is determined optimally to decrease the objective (up to the first order) among all rank-drop steps explicitly, we believe that this is a better optimization formulation when the iterate approaches the boundary.

3.3.2 Solving the Optimization Problem in the Interior Case

Now we discuss how to solve the Rank-Drop optimization problem (12) in the interior case. Let $W\coloneqq U^{\top}\nabla f(X_{k})V$.

The Lagrangian for (12) is,

[TABLE]

Theorem 3.12**.**

Suppose $(s,t,\lambda,\alpha,\beta)$ satisfies the KKT conditions of (12) with $\lVert t\rVert_{2}<1$. Then $\alpha=\beta=0$, $\lambda$ is an eigenvalue of $-\Sigma W$, and $M_{\lambda}\coloneqq-\frac{1}{2}(W+\lambda\Sigma^{-1})$ has zero as a singular value. Conversely, assume that $(\hat{s},\hat{t})$ forms a singular vector pair of $M_{\lambda}$ associated with the singular value zero and $\kappa(X_{k})\hat{s}^{\top}\Sigma^{-1}\hat{t}>1$, then the KKT conditions of (12) are satisfied at $(\hat{s},\hat{t}/(\kappa(X_{k})\hat{s}^{\top}\Sigma^{-1}\hat{t}),\lambda,0,0)$.

Since problem (12) is nonconvex and generally difficult to solve, we only consider candidate KKT points characterized in Theorem 3.12. For each eigenvalue $\lambda$ of $-\Sigma Wx$, we fix $\alpha=\beta=0$ and choose $(\hat{s},\hat{t})$ that corresponds to the singular vector pair of $-0.5(W+\lambda\Sigma^{-1})$ with the singular value zero. If $(s,t)=(\hat{s},\hat{t}/(\kappa(X_{k})\hat{s}^{\top}\Sigma^{-1}\hat{t}))$ is feasible with respect to (12) and $\lVert t\rVert_{2}<1$, then by Theorem 3.12, $(s,t,\lambda,0,0)$ satisfies the KKT conditions.

In the event that no feasible candidate is found, we solve (15) instead and obtain a feasible rank-drop step, guaranteed by Theorem 3.9. We remark that, although it is always possible to find feasible rank-drop steps using (15), in the interior case, it is still preferable to solve problem (12) since the assumption that $s=t$ can be restrictive and unnecessary when far away from the boundary. Algorithm 3 presents the details of the rank-drop direction computation.

3.4 Step Selection Criteria

The criteria in Freund et al. (2015) for away-steps require solving the regular FW linear subproblem at each iteration even when the FW step is not taken. In the rank-drop framework, we accept any rank-drop step that does not increase the objective. This allows the algorithm to maintain a lower rank SVD as well as allows the algorithm to skip computing a FW step unnecessarily. Additionally, for the rank-drop step, we observe that it is very rare that the algorithm chooses a rank-drop step two iterations in a row. Thus, we only generate the rank-drop direction when the previous iterate is a regular FW step, avoiding unnecessary rank-drop step computations.

4 Convergence Analysis

Following the proof for Theorem 4 in Guélat and Marcotte (1986), we show that the iterates, from Rank Drop FW in Algorithm 4, converge to the global optimum of (1).

Theorem 4.1**.**

Let $\{X_{k}\}$ be a sequence generated by Algorithm 4 and let $f^{*}$ be the optimal value for problem (1). Assume $\nabla f(X)$ is Lipschitz continuous in the feasible region. Then $f(X_{k})-f^{*}\leq\frac{8\delta^{2}L}{4+N_{fw}^{k}}$, where $N_{fw}^{k}$ be the number of FW steps taken up to the iteration $k$.

5 Complexity Per Iteration

When computing the rank-drop steps, we note that the dimension of the subproblems is $r$, the rank of the current iterate. First, forming the matrix $W\coloneqq U^{\top}\nabla f(X_{k})V$ requires $O(r^{2}h\min\{m,n\})$ operations, where $h$ is the maximum number of nonzero elements in any row or column. In the interior case, we must compute an eigen-decomposition of an $r\times r$ matrix which takes $O(r^{3})$ time. Then, each eigenvalue $\lambda$ is used to form the matrix $-0.5(W+\lambda\Sigma^{-1})$ where the singular vector pair corresponding to the zero singular value is computed. The total time required for the interior case is $O(r^{3}+r^{2}h\min\{m,n\})$. In the exterior case, $O(r^{2})$ flops are required to compute the largest eigenvalue. Thus, the total complexity per iteration is $O(r^{3}+r^{2}h\min\{m,n\})$.

6 Experimental Results

We validate the rank-drop steps on a matrix completion task using various datasets from MovieLens222http://grouplens.org/datasets/movielens/. We first center and scale each data set to have mean 0 and standard deviation 1.

We compare the proposed Rank-Drop Frank Wolfe (RDFW) against the aforementioned FW variants, FW Frank and Wolfe (1956), AFW Lacoste-Julien and Jaggi (2015), and IF$(0,\infty)$, Freund et al. (2015), as well as a state-of-the-art nuclear norm regularized solver in ActiveALT Hsieh and Olsen (2014).

Following Yao et al. (2016), we randomly partition each dataset into 50% training, 25% validation, and 25% testing. The $\delta$ value in (1) is tuned with $\delta=\mu_{j}\cdot\lVert Y\rVert_{F}$, where $\lVert Y\rVert_{F}$ is the Frobenius norm of the training data matrix, and $\mu_{j}=2+0.2j,j\in\mathbb{N}$. We increase $j$ until the mean RMSE on the validation set does not improve by more than $10^{-3}$. We terminate the algorithm when an upper bound on the relative optimality gap ensures $(f(X_{k})-f^{*})/f^{*}<10^{-2}$ or a maximum iteration count of 1000 is reached.

For ActiveALT, a regularized nuclear norm problem (not constrained) is solved where the regularization parameter $\lambda$ is chosen by approximately solving for the Lagrange multiplier from the solution to the constrained problem. From the optimality conditions, we have $U^{\top}\nabla f(X^{*})V+\lambda I=0$. Thus, $\lambda$ is approximated by the mean of the diagonal values of $U^{\top}f(X_{k})V$, where $X_{k}$ is the converged solution of RDFW. ActiveALT terminates when $f(X_{k-1})-f(X_{k})<10^{-4}$, to match with the criterion suggested in Yao et al. (2016) with a maximum iteration count of 150.

6.1 Computational Details

All simulations were run in MATLAB. For all FW variants, we maintain a thin SVD for the current iterate, where the SVD is updated at each iteration using a rank-one update as described in Brand (2006). The rank is calculated by counting all singular values larger than $10^{-6}$, in agreement with the definition of rank in Freund et al. (2015).

7 Discussion

We observe that the proposed method, RDFW, shows large improvements over FW and its common variants in terms of the rank and computing time. While ActiveALT, the leading nuclear norm based matrix completion solver, is very competitive with RDFW, RDFW has the additional attractive property that it does not require knowledge of the structure of the Hessian, or even require $f(\cdot)$ to be twice differentiable, allowing for greater generality. Moreover, since the matrix completion objective is quadratic, this is the ideal situation for ActiveALT since the underlying solver in ActiveALT only requires one Newton step to converge, whereas with a nonquadratic objective, the computational burden can increase.

8 Conclusions

We have proposed a rank-drop optimization formulation to determine optimally descent rank-drop steps for the nuclear-norm constrained minimization. By considering the interior and exterior cases separately to ensure feasibility, we also devise subproblems that can be efficiently solved. The proposed formulation can be deployed in a projection free minimization method, e.g., FW method, to efficiently compute a low rank solution by maintaining low rank intermediate iterates, without compromising the strong convergence guarantees. While classic FW methods tend to have very high rank solutions for nuclear-norm constrained problems, we have shown that the addition of rank-drop steps can drastically reduce the rank of the iterates, allowing for much faster algorithms to reach low rank solutions with less required space.

Acknowledgements

Authors acknowledge funding from the National Sciences and Engineering Research Council of Canada.

Appendix A Omitted Proofs

A.1 Proof of Lemma 1

Proof.

Assume that $\sigma^{-1}uv^{T}\in\mathcal{R}(A)$. Then $u=Ax$ and $v=A^{\top}y$ for some $x$ and $y$ and $u,v\neq 0$. Let $A=U\Sigma V^{\top}$ be a thin SVD and write $u=Us$ and $v=Vt$ for some $s,t\in\mathbb{R}^{r}$.

We have,

[TABLE]

Consequently,

[TABLE]

Finally, we note that $t^{\top}\Sigma^{-1}s$ can always be made positive by replacing $s$ with $-s$. ∎

A.2 Proof of Lemma 3.3

Proof.

Let $Z=\sigma^{-1}uv^{\top}$. Without loss of generality, we can assume that $\sigma^{-1}>0$ since we can always substitute $u$ with $-u$ to make $\sigma^{-1}$ positive. Let $\alpha=\sigma^{-1}\lVert u\rVert_{2}\lVert v\rVert_{2}$. We need to show that $\alpha\geq\sigma_{r}(A)$.

From Lemma 1, we know that $\sigma^{-1}uv^{\top}=\frac{Ust^{\top}V^{\top}}{s^{\top}\Sigma^{-1}t}$ for some $s,t\in\mathbb{R}^{r}$. Then,

[TABLE]

∎

A.3 Proof of Theorem 3.4

Proof.

Define $R_{k}=X_{k}-Z_{k}$. Note that,

[TABLE]

where the last inequality uses $\lVert Z_{k}\rVert_{NN}\leq(\delta-\lVert X_{k}\rVert_{NN})/2$.

Let $R_{k}$ have the thin SVD, $R_{k}=\sum_{i=1}^{r-1}\sigma_{i}u_{i}v_{i}^{\top}$. We can express $X_{k}$ as,

[TABLE]

where $\alpha_{0}=\lVert Z_{k}\rVert_{NN}/\delta$ and $\alpha_{i}=\sigma_{i}/\delta$ for $i\in\{1,...,r-1\}$. Note that $\alpha_{i}\geq 0,~{}i=0,\ldots,r-1$, and

[TABLE]

Next, we wish to show that $X_{k+1}\in\mathcal{B}_{NN}(0,\delta)$. Rewrite $X_{k+1}$ as,

[TABLE]

where $\beta_{0}=(\alpha_{0}+\alpha_{0}\tau-\tau)$ and $\beta_{i}=(1+\tau)\alpha_{i},\forall i\in\{1,...,r-1\}$.

If $\tau\in[0,\tau^{*}]$, then $\alpha_{0}+\alpha_{0}\tau-\tau\geq 0$, with equality at $\tau=\tau^{*}$. Thus, $\beta_{i}\geq 0,\forall i\in\{0,1,...,r-1\}$. Next, we have that,

[TABLE]

Thus, $\beta_{i}\geq 0,~{}i=0,\ldots,r-1$, and $\sum_{i=0}^{r-1}\beta_{i}\leq 1$. This implies that $X_{k+1}\in\mathcal{B}(0,\delta)$ if $\tau\in[0,\tau^{*}]$.

Finally, when $\tau=\tau^{*}$, we have that $(\alpha_{0}+\alpha_{0}\tau^{*}-\tau^{*})=0$. Hence, $X_{k+1}$ is a sum of $r-1$ rank one matrices and $rank(X_{k+1})\leq r-1$. Since $X_{k+1}$ is a rank one perturbation of $X_{k}$, the most the rank can decrease by is 1, thus $rank(X_{k+1})=r-1$. ∎

A.4 Proof of Theorem 3.5

Proof.

Assume that $X_{k}$ has the thin SVD $X_{k}=U\Sigma V^{T}$. We show that the smallest singular vector pair can be made into a rank-drop step. Let $s=e_{r}$, where $e_{r}$ is the $r$th elementary basis vector, $t=(\sigma_{r}(X_{k})/\kappa(X_{k}))e_{r}$, and $Z=Ust^{\top}V^{\top}/s^{\top}\Sigma^{-1}t$. Since $\sigma_{r}(X_{k})\leq\kappa(X_{k})$, $\lVert t\rVert_{2}\leq 1$. We conclude that,

[TABLE]

Thus $Z\in\mathcal{B}_{NN}(0,\kappa(X_{k}))\cap\mathcal{R}(X_{k})$, and the feasible region to (10) is non-empty. More generally, we can see that any singular vector pair with singular value $\sigma_{i}(X_{k})\leq\kappa(X_{k})$ will be feasible to (10) as well. ∎

A.5 Proof of Theorem 3.6

Proof.

First we will show that for any feasible solution $(s,t)$ to (11), there exists a corresponding feasible solution to (12) with the same objective value.

Let $(s,t)$ be a feasible solution to (11). Define $\hat{s}\coloneqq\frac{s}{\lVert s\rVert_{2}}\text{ and }\hat{t}\coloneqq\frac{\kappa(X_{k})^{-1}}{\hat{s}^{\top}\Sigma^{-1}t}t$. We will show $(\hat{s},\hat{t})$ is feasible for (12) and the objective values of (11) and (12) are equal at $(s,t)$ and $(\hat{s},\hat{t})$. To see this, note that,

[TABLE]

But,

[TABLE]

satisfying the first constraint of (12). We can also conclude,

[TABLE]

showing the objective values will be equal.

For the norm constraint, $\lVert\hat{s}\rVert_{2}=1$ by construction. Next, to see that the solution satisfies the last constraint, the fact that $(s,t)$ is feasible gives us,

[TABLE]

where the first implication uses the result from (23). Next, using the fact that the nuclear norm of a rank-one matrix is equivalent to its Frobenius norm, we have,

[TABLE]

Thus, the results from (24) and (25) jointly imply that $\lVert\hat{t}\rVert_{2}\leq 1$. It is readily seen that any feasible solution to (12) is a feasible solution to (11). Since there exists a mapping from feasible points in (11) to (12) and vice versa preserving objective values, the optimal values must be equal. Thus, an optimal solution to (12) is an optimal solution to (11) and the converse result holds as stated. ∎

A.6 Proof of Theorem 3.7

Proof.

From Watson [1992], the subdifferential of the nuclear norm is,

[TABLE]

Let $G\in\partial\lVert X_{k}\rVert_{NN}$ be an arbitrary subgradient. Then,

[TABLE]

This implies that $\max_{G\in\partial\lVert X\rVert_{NN}}\langle D_{k},G\rangle\leq 0$ if and only if $\delta s^{\top}t\geq\lVert X_{k}\rVert_{NN}$. Since $D_{k}$ has a negative inner product with all elements in the subdifferential, it must be a descent direction for the nuclear norm at $X_{k}$. ∎

A.7 Proof of Corollary 3.7.1

Proof.

From Theorem 3.7, we must have that $\delta s^{\top}t\geq\lVert X_{k}\rVert_{NN}=\delta$. This implies that $s^{\top}t\geq 1$. Since $\lVert s\rVert_{2}=\lVert t\rVert_{2}=1$, we have that $s^{\top}t\leq 1$, where equality is attained only when $s=t$. Thus, $\delta s^{\top}t\geq\lVert X_{k}\rVert_{NN}$ if and only if $s=t$, completing the proof. ∎

A.8 Proof of Lemma 3.8

Proof.

Note that $\sigma_{r}(X_{k})<\lVert X_{k}\rVert_{NN}/r$. Then, the inequality can be rearranged as follows.

[TABLE]

∎

A.9 Proof of Theorem 3.9

Proof.

From the definition of $X_{k+1}=(1+\tau^{*})U\Sigma V^{\top}-\tau^{*}\delta Uss^{\top}V^{\top}=U((1+\tau^{*})\Sigma-\tau^{*}\delta ss^{\top})V^{\top}$. Let $M=(1+\tau^{*})\Sigma-\tau^{*}\delta ss^{\top}$. It is clear that $rank(X_{k+1})=rank(M)$. From Theorem 3.1, it is straightforward to verify that $\tau^{*}\delta ss^{\top}\in\mathcal{R}((1+\tau^{*})\Sigma)$, so $rank(M)=rank(X_{k})-1$.

Let $\lambda_{i}(M)$ be the $i\textsuperscript{th}$ eigenvalue of $M$, then,

[TABLE]

We require the following Theorem from Chu et al. [1998].

Theorem A.1** (Chu et al. [1998]).**

*Suppose that $D$ is symmetric positive semidefinite, $S$ is symmetric, and

$rank(D-S)=rank(D)-rank(S)$. Then $D-S$ is positive semidefinite.*

From Theorem A.1, $M$ is symmetric positive semidefinite, so $\sum_{i}\lambda_{i}(M)=\sum_{i}\sigma_{i}(M)\Rightarrow\lVert M\rVert_{NN}\leq\delta$. Since $X_{k+1}=UMV^{\top}$, it follows that $\lVert X_{k+1}\rVert_{NN}\leq\delta$. ∎

A.10 Proof of Lemma 3.10

Proof.

If $\kappa(X_{k})>\sigma_{r}(X_{k})$, then $\lVert X_{k}\rVert_{NN}<\delta$, and $\mathcal{F}(X_{k})=\mathcal{B}_{NN}(0,\delta)$. Then $Z=\kappa(X_{k})Ust^{\top}V^{\top}$ where $\lVert s\rVert_{2}=1$ and $\lVert t\rVert_{2}\leq 1$. This implies that $\lVert Z\rVert_{NN}\leq\kappa(X_{k})<\delta$ and $Z\in\mathcal{F}(X_{k})$. ∎

A.11 Proof of Lemma 3.11

Proof.

If $s$ is a solution to (15), then $\lVert s\rVert_{2}=1$ and $\lVert Z\rVert_{NN}\leq\delta$. If $\lVert X_{k}\rVert_{NN}<\delta$, then $Z\in\mathcal{B}_{NN}(0,\delta)=\mathcal{F}(X_{k})$.

Otherwise, $\lVert X_{k}\rVert_{NN}=\delta$ and $\mathcal{F}(X_{k})=\{UMV^{\top}:M\succeq 0,\text{tr}(M)=\delta\}$, as defined in (4). Clearly, $\delta ss^{\top}$ is symmetric positive semidefinite, and since $\text{tr}(\delta ss^{\top})=\delta s^{\top}s=\delta$, we can set $M=\delta ss^{\top}$ and $Z=UMV^{\top}\in\mathcal{F}(X_{k})$. ∎

A.12 Proof of Lemma 3.12

Proof.

Since $(s,t,\lambda,\alpha,\beta)$ satisfies the KKT conditions, we have,

[TABLE]

Let $M_{\lambda}=-\frac{1}{2}(W+\lambda\Sigma^{-1})$. Then we can rewrite (27) as,

[TABLE]

Following these equations, we conclude that $\alpha=s^{\top}M_{\lambda}t=\beta\lVert t\rVert^{2}_{2}$ for any feasible solution, since $s^{\top}s=1$.

Since $(s,t,\lambda,\alpha,\beta)$ satisfies the KKT conditions, from the complementary slackness and the additional assumption $t^{\top}t<1$, we must have $\beta=0$. This implies that $0=\beta=\alpha$, since $\alpha=\beta\lVert t\rVert^{2}$. Thus, $M_{\lambda}$ must be rank deficient. In other words, there exists $\lambda\in\mathbb{R}$ and a vector $x\in\mathbb{R}^{r}$ such that,

[TABLE]

Hence $\lambda$ is an eigenvalue of $-\Sigma W$.

With $\alpha=\beta=0$, for each real $\lambda\in\text{eigs}(-\Sigma W)$, (27) is satisfied by setting $\hat{s}$ and $\hat{t}$ to any singular value pair of $M_{\lambda}$ with associated singular value of 0. Conversely, assume that $(\hat{s},\hat{t})$ forms a singular vector pair of $-\frac{1}{2}(W+\lambda\Sigma^{-1})$ associated with the singular value zero, and $\kappa(X_{k})\hat{s}^{\top}\Sigma^{-1}\hat{t}>1$. Then $(s,t)=(\hat{s},\hat{t}/(\kappa(X_{k})\hat{s}^{\top}\Sigma^{-1}\hat{t}))$ is a feasible point of (12) and $(s,t,\lambda,0,0)$ satisfies the KKT condition of (12). Thus, we have characterized all possible points that can satisfy the KKT conditions such that $t^{\top}t<1$. ∎

A.13 Proof of Theorem 4.1

Proof.

We will use a similar proof as Guélat and Marcotte [1986]. Since rank-drop steps always decrease the rank of the solution, the number of rank-drop steps is bounded by the number of Frank-Wolfe steps. Thus, any sequence $\{X_{k}\}$ contains an infinite number of Frank-Wolfe steps. Since rank-drop steps can only decrease the objective, the convergence is guaranteed by the same arguments as the regular Frank-Wolfe algorithm. ∎