Chordal Decomposition in Rank Minimized Semidefinite Programs with Applications to Subspace Clustering

TL;DR

This paper introduces a novel chordal decomposition approach for rank-constrained SDPs, enabling efficient solutions for problems like subspace clustering by decomposing large rank constraints into smaller ones.

Contribution

It develops a method to decompose rank constraints in SDPs using chordal sparsity and a re-weighted heuristic, improving computational efficiency for large-scale problems.

Findings

Significant speed-up in solving rank-minimized SDPs.

Effective application to subspace clustering tasks.

Preservation of sparsity pattern across iterations.

Abstract

Semidefinite programs (SDPs) often arise in relaxations of some NP-hard problems, and if the solution of the SDP obeys certain rank constraints, the relaxation will be tight. Decomposition methods based on chordal sparsity have already been applied to speed up the solution of sparse SDPs, but methods for dealing with rank constraints are underdeveloped. This paper leverages a minimum rank completion result to decompose the rank constraint on a single large matrix into multiple rank constraints on a set of smaller matrices. The re-weighted heuristic is used as a proxy for rank, and the specific form of the heuristic preserves the sparsity pattern between iterations. Implementations of rank-minimized SDPs through interior-point and first-order algorithms are discussed. The problem of subspace clustering is used to demonstrate the computational improvement of the proposed method.

Equations

$$X^{\ast }=\underset{X}{\mathrm{argmin}}$$

$$⟨A_i,X⟩=b_i,i=1,\dots ,m,$$

$$X⪰0,\text{rank}(X)\le t,$$

$$E_{{\mathcal{C}}_k}X{E}_{{\mathcal{C}}_k}^T\in {\mathbb{S}}_{+}^{|{\mathcal{C}}_k|},k=1,\dots ,p$$

$$\text{rank}(X)=\underset{k}{\max }\text{rank}(E_{{\mathcal{C}}_k}X{E}_{{\mathcal{C}}_k}^T)$$

$$\underset{X,X_k}{\min }$$

$$⟨A_i,X⟩=b_i,i=1,\dots ,m$$

$$X_k=E_{{\mathcal{C}}_k}X{E}_{{\mathcal{C}}_k}^T,$$

$$X_k\in {\mathbb{S}}_{+}^{|{\mathcal{C}}_k|},\forall k=1,\dots ,p,$$

$$\underset{X}{\min }$$

$$⟨A_i,X⟩=b_i,i=1,\dots ,m,$$

$$X\in {\mathbb{S}}_{+}^n(\mathcal{E},?).$$

$$\underset{r,s}{\min }$$

$$s_{i,j}|r_i^T{x}_j|\le s_{i,j}ϵ,{\forall }_{i=1}^{N_s},{\forall }_{j=1}^{N_p},$$

$$s_{i,j}=s_{i,j}^2,{\forall }_{i=1}^{N_s},{\forall }_{j=1}^{N_p},$$

$$\sum _{i=1}^{N_s}{s}_{i,j}=1,{\forall }_{j=1}^{N_p},$$

$$r_i^T{r}_i=1,{\forall }_{i=1}^{N_s},$$

Full text

Chordal Decomposition in Rank Minimized SDPs

Jared Miller\inst1, Yang Zheng\inst2, Biel Roig-Solvas\inst1, Mario Sznaier\inst1, Antonis Papachristodoulou\inst3

\setbeamertemplate

caption[numbered]

\institute[shortinst]\inst1 Northeastern University \samelineand\inst2 Harvard University \samelineand\inst3 University of Oxford

{columns}

[t] {column}

{column}

{block}

Background Semidefinite Programs (SDPs) are convex optimization problems that often arise in machine learning as relaxations of NP-hard problems. The SDP relaxation of the original problem is tight if the output $X$ obeys certain rank conditions. Imposing the non-convex rank constraint on the SDP:

[TABLE]

Solving an SDP scales polynomially with the size of the PSD constraint, and the rank constraint worsens performance. Chordal decompositions break up the large PSD constraint into a product of smaller ones, and use a rank completion result to efficiently reduce $\textrm{rank}(X)$. We apply this scheme to subspace clustering, and present an algorithm that is linear in the number of datapoints and subspaces.

{block}

Chordal Graphs and Semidefinite Optimization In many SDPs, only a small subset of entries of $X$ are found in $(C,A_{i})$. All other entries can be completed arbitrarily to force $X\succeq 0$. These SDPs are ‘chordally sparse’.

Let $\mathcal{G}(\mathcal{V},\mathcal{E})$ be a graph formed by stacking $(C,A_{i})$ into an adjacency matrix. $\mathcal{G}$ is chordal if all $\geq 4$-length cycles have shortcuts. Cliques are subsets of vertices that are strongly connected. Maximal cliques of a chordal A chordal graph has at most $\lvert\mathcal{V}\rvert$ maximal cliques, which can be found in linear time. Efficient heuristics exist to extend non-chordal graphs to chordal ones with a minimal number of edges.

The cone $\mathbb{S}_{+}^{n}(\mathcal{E},?)$ is the set of matrices with sparsity pattern $\mathcal{E}$ that can be completed to be PSD. Let $\{\mathcal{C}_{k}\}$ be the set of maximum cliques of $\mathcal{G}$, and $E_{\mathcal{C}_{k}}$ index out values in $X$ involved in $\mathcal{C}_{k}$. Two theorems allow for efficient optimization:

\heading

Grone’s theorem [grone1984positive] Let $\mathcal{G}(\mathcal{V},\mathcal{E})$ be a chordal graph with a set of maximal cliques $\{\mathcal{C}_{1},\mathcal{C}_{2},\ldots,\mathcal{C}_{p}\}$. Then, $X\in\mathbb{S}^{n}_{+}(\mathcal{E},?)$ iff

[TABLE]

\heading

Minimum Rank Completion [dancis1992positive] For any $X\in\mathbb{S}^{n}_{+}(\mathcal{E},?)$, there exists at least one minimum rank PSD completion where

[TABLE]

Minimizing the rank of $X$ is therefore equivalent to minimizing the maximum rank among all $X_{k}$.

{block}

Rank Relaxations

The Rank constraint is non-convex, but there exist convex surrogates for approximation. The nuclear norm $\lVert X\rVert_{*}=\sum_{i}{\sigma_{i}(X)}$, and $\lVert X\rVert_{*}=\textrm{Tr}(X)$ when $X\succeq 0$. The nuclear norm equally weights all singular values, and generally fails to find a minimal rank solution. The reweighted heuristic adds a weighting term $\lVert W^{t}X\rVert_{*}$ in each iteration $t$, where $W^{t}=(X^{t-1}+\delta I)^{-1}$ [mohan2010reweighted]. This adds a higher penalty to low singular values and encourages sparsity.

{column}

{column}

{block}

Chordal Decomposition of Rank-Minimized SDP

The chordalized rank-minimized SDP is:

[TABLE]

{block}

Algorithmic Implementation Problem (2) is convex for each reweighting iteration ($W_{k}$ update). Tests were run on Subspace Clustering and Maxcut SDPs. \headingInterior Point Method Interior point methods (such as SeDuMi, MOSEK, SDPT3) will generally suffer under the additional equality constraints. Eliminating $X_{k}$ forms:

[TABLE]

where $W_{\mathcal{C}}=\sum_{k=1}^{p}E_{\mathcal{C}_{k}}^{T}W_{k}E_{\mathcal{C}_{k}}$ is the accumulated clique weight. The reweighted cost $C+W_{\mathcal{C}}$ retains the existing sparsity pattern $\mathcal{E}$. Decomposition methods such as SparseCoLo [fujisawa2009user] will leverage chordal sparsity in the interior point algorithm.

\heading

First Order Algorithms Problem (2) is well suited for ADMM by a variable split on $(X,X_{k})$ [boyd2011distributed]. This forms a 3 step process in each iteration: solve a quadratic program in $X$, project onto PSD cones for $X_{k}$, and then dual ascend on multipliers $\Lambda_{k}$.

{column}

{column}{block}

Subspace Clustering

Given $N_{p}$ points $x_{j}$ in a $D$-dimensional space and a $N_{s}$ subspaces with normals $r_{i}$, subspace clustering aims to find binary labels $s_{ij}$ to determine whether point $x_{j}$ came from subspace $r_{i}$.

$x_{j}$ came from $r_{i}$ if $r_{i}^{T}x_{j}=0$, relaxed under bounded noise to $\lvert r_{i}^{T}x_{j}\rvert\leq\epsilon$. Solving for $r_{i}$ and $s_{ij}$ is a nonconvex quadratic feasibility problem:

[TABLE]

This can be cast as a rank-1 SDP in $X=[1,v][1,v]^{T}$ where $v=[r_{i};s_{ij}]$, where $X$ has size $(1+N_{s}(D+N_{p}))$. (4) exhibits chordal sparsity, as terms such as $s_{i_{1}j_{1}}s_{i_{2}j_{2}}$ or $r_{1_{1}}r_{2_{1}}$ do not appear. [cheng2016subspace] noticed this sparsity solved this problem through the grey chordal extension below, and we improve this by using a reduced chordal extension (red).

Let $N[k]\cong(\mathbb{S}_{+}^{k})^{N}$ denote $N$ different $k\times k$ PSD cones. A summary of the matrix sizes are:

[TABLE]

\heading

Further Information

{column}