Restricted Connection Orthogonal Matching Pursuit For Sparse Subspace Clustering

TL;DR

The paper introduces RCOMP-SSC, a noise-robust, computationally efficient clustering algorithm that improves accuracy by restricting data point connections during Orthogonal Matching Pursuit, validated on synthetic and real datasets.

Contribution

It proposes a novel RCOMP-SSC algorithm that enhances clustering accuracy and efficiency by restricting connections, and introduces a scalable control matrix framework for data point selection strategies.

Findings

Outperforms existing methods in accuracy on synthetic and real data

Reduces computational time compared to traditional SSC

Demonstrates robustness to noise in clustering tasks

Abstract

Sparse Subspace Clustering (SSC) is one of the most popular methods for clustering data points into their underlying subspaces. However, SSC may suffer from heavy computational burden. Orthogonal Matching Pursuit applied on SSC accelerates the computation but the trade-off is the loss of clustering accuracy. In this paper, we propose a noise-robust algorithm, Restricted Connection Orthogonal Matching Pursuit for Sparse Subspace Clustering (RCOMP-SSC), to improve the clustering accuracy and maintain the low computational time by restricting the number of connections of each data point during the iteration of OMP. Also, we develop a framework of control matrix to realize RCOMP-SCC. And the framework is scalable for other data point selection strategies. Our analysis and experiments on synthetic data and two real-world databases (EYaleB & Usps) demonstrate the superiority of our algorithm compared with other clustering methods in terms of accuracy and computational time.

Figures

Tables

Table 1. Table 1 : Accuracy (%)/Computational time (s) on Usps

	200	400	600	800	1000
RCOMP	69.56/	72.37/	72.08/	71.77/	70.00/
(ours)	0.22	0.60	1.27	2.66	5.02
OMP	62.52/	61.21/	56.17/	55.65/	55.79/
	0.08	0.18	0.33	0.53	0.75
AOMP	67.75/	70.20/	66.62/	67.84/	67.65/
	1.34	5.04	11.4	20.5	32.0
ROMP	68.10/	69.90/	66.92/	66.24/	66.37/
	2.02	6.92	15.8	27.9	43.9
SSC	61.24/	60.28/	60.81/	60.72/	61.36/
	6.54	37.8	113	258	515
ENSC	59.56/	57.97/	54.97/	52.75/	52.22/
	37.4	98.4	106	243	328
LRR	62.04/	64.54/	63.84/	65.30/	66.68/
	5.86	14.7	31.4	56.6	101

Table 2. Table 2 : Accuracy (%) on EYaleB

	5	10	20	30	38
RCOMP(ours)	96.67	94.77	89.20	86.00	83.33
OMP	95.24	86.77	81.57	77.60	76.59
AOMP	95.23	88.15	82.26	79.58	77.96
ROMP	96.52	93.94	85.92	81.29	77.96
SSC	83.13	58.31	56.36	54.75	57.66
ENSC	76.47	76.19	64.74	60.56	56.53
LRR	67.37	76.60	69.50	66.26	64.08

Equations

$${\mathbf{x}}_i={\mathbf{Xc}}_{i}s.t.c_{ii}=0$$

$${\mathbf{c}}_i={\mathrm{argmin}}_{{\mathbf{c}}_i}\Vert {\mathbf{c}}_i{\Vert }_1+\lambda \Vert {\mathbf{x}}_i-{\mathbf{Xc}}_i{\Vert }_{2}s.t.c_{ii}=0$$

$$\mathbf{W}=|\mathbf{C}|+|\mathbf{C}{|}^T$$

$${\mathbf{c}}_i={\mathrm{argmin}}_{{\mathbf{c}}_i}\Vert {\mathbf{x}}_i-{\mathbf{Xc}}_i{\Vert }_2^{2}s.t.\Vert {\mathbf{c}}_i{\Vert }_0\le k,c_{ii}=0$$

$${\mathbf{dot}}^{\prime }=\{{\mathbf{dot}}_i^{\prime }|do{t}_{ij}^{\prime }=do{t}_{ij}\times M_{ij}\}$$

$$\{{\mathbf{x}}_j|\exists {\mathbf{r}}_i\in S_i,{\mathbf{x}}_j\notin S_i,\forall {\mathbf{x}}_i\in S_i,|{\mathbf{r}}_i{\mathbf{x}}_i|<|{\mathbf{r}}_i{\mathbf{x}}_j|\}$$

Full text

**Index Terms— ** Restricted Connection, Noise Robust, Orthogonal Matching Pursuit (OMP), Sparse Subspace Clustering (SSC)

1 Introduction

The unsupervised learning methods such as Subspace Clustering(SC) [1] that can learn from the unlabeled data have become more and more important. SC can be applied in many computer vision tasks in which high dimensional data can be approximated as a union of low dimensional subspaces [1, 2], including face clustering [3], image representation [4], motion segmentation [5], and written digit clustering [6].

There are several kinds of SC methods, including algebraic, iterative, statistical, and spectral clustering-based methods [1]. Further the spectral clustering-based methods are divided into different categories according to the norm regularization they choose for sparse coefficients. For instance, Low Rank Representation (LRR) [7, 8] based on nuclear norm represents the data points with the lowest-rank representation among all the candidates; Least Square Regression (LSR) [9] with $\ell$2 norm groups the highly correlated data; Exemplar-based Subspace Clustering (ESC) [10] based on $\ell$1 norm focuses on the class-imbalanced data; Elastic Net Subspace Clustering (ENSC) [11] adopts both $\ell$1 and $\ell$2 norm to find better coefficients; Subspace Learning by $\ell$0-Induced Sparisty [12] employs proximal gradient descent to obtain a sub-optimal solution; while Sparse Subspace Clustering (SSC) [13] with the $\ell$1 norm calculates a sparse self-representation of data points. And Structured Sparse Clustering [14] learns both affinity and segmentation of SSC.

SSC is inefficient when analyzing the large-scale dataset, so here come the Sparse Subspace Clustering by Orthogonal Matching Pursuit (SSC-OMP) [15] and Subspace Clustering via Matching Pursuit (SSC-MP) [16], which are much faster than SSC. However, SSC-OMP may suffer from a reduction of accuracy at the presence of noise[17].

In order to improve the accuracy of OMP, Active Orthogonal Matching Pursuit for Sparse Subspace Clustering (AOMP-SSC) [18] updates the data and drops them randomly in the process of OMP, while Sparse Subspace Clustering by Rotated Orthogonal Matching Pursuit (SSC-ROMP) [19] rotates the data points. Both AOMP-SSC and ROMP-SCC have raised the computational complexity of SSC-OMP, while the accuracy is not improved enough. Moreover, they both change the data points in the process of OMP, which disrupts the original distribution of the data points.

In this paper, we propose the concept of connection and find that the number of connections of a data point is the potential sparsity of it when it comes to the affinity matrix, which affects the clustering result. Therefore, we propose a noise-robust Restricted Connection Orthogonal Matching Pursuit for Sparse Subspace Clustering (RCOMP-SSC) to address the issue of accuracy reduction of OMP and keep the original form of the data points, while still enjoying the low computational complexity. Also, we develop a framework of data point selection with control matrix of the inner products of the residuals and the other data points, on which we can realize RCOMP. More importantly, the control matrix is scalable, since most of the data point selection strategies, including AOMP-SSC and ROMP-SSC, can act on it. We demonstrate through experiments on different tasks of computer vision, i.e., clustering images of faces under varying illumination conditions (EYaleB) and hand written digit clustering (Usps) that RCOMP-SSC outperforms other methods in terms of accuracy and computational time.

2 PRELIMINARY

Given a dataset X = [x1, x2 ,…, xN] $\in$ $\mathbb{R}$D×N, where every x${}_{\emph{i}}$ is a data point of $\ell$2 norm. D stands for the original dimension of the data. And there are totally N data points in the dataset. SC clusters data points into their original subspaces{S${}_{\emph{i}}$}${}^{n}_{\emph{i}=1}$with the dimensions{d${}_{\emph{i}}$}${}^{n}_{\emph{i}=1}$. That is,

[TABLE]

2.1 Sparse Subspace Clustering (SSC)

SSC calculates the coefficient matrix CSSC={c${}_{\emph{i}}$}${}^{N}_{\emph{i}=1}$ in (2) by $\ell$1 norm, which aims at finding a sparse representation of each data point. Every c${}_{\emph{i}}$ is correspond to x${}_{\emph{i}}$ , of which the nonzero entries indicate the data points from the same subspace. And the last part measures the similarity between x${}_{\emph{i}}$ and X****c${}_{\emph{i}}$ when there is noise.

[TABLE]

After calculating the coefficient matrix, SSC applies the spectral clustering [20] to the affinity matrix below:

[TABLE]

2.2 Sparse Subspace Clustering by Orthogonal Matching Pursuit(SSC-OMP)

SSC-OMP finds k neighbors of x${}_{\emph{i}}$ and represents x${}_{\emph{i}}$ with its neighbors (4). In OMP, another data point {x${}_{\emph{j}}$ s.t. j$\neq$i} that has the largest absolute inner product with the current residual will be added to the neighbor set of x${}_{\emph{i}}$. Then x${}_{\emph{i}}$ is iteratively projected onto the span of its current neighbors so as to update the residual (r) until k neighbors of x${}_{\emph{i}}$ is found or the residual is small enough.

[TABLE]

3 RCOMP-SCC

In this section, we first illustrate the inspiration behind our proposed algorithm. And then we demonstrate the detailed procedure of RCOMP-SSC. The analysis of the algorithm is showed at the last part.

3.1 Intuition

Definition 1** (connection)**

A connection is formed from xi to xj when xi choose xj as its neighbor.

In OMP, the sparsity of the representation of xi is definite as the parameter k is settled, which is exactly equal to k. However, the clustering result relies on the structure of W in (3), which computes not only the connections between xi and its neighbors but also the connections between xi and {xj s.t. j $\neq$ i} who choose xi as a neighbor. We define the total number of the two kinds of connections of one data point as con, and the sparsity of xi is represented by con rather than k when it comes to the affinity matrix. Notice that the total number of con in affinity matrix is constant as equals to 2Nk. However, the con is not equal for each data point, since some data points have large ones for being chosen as neighbors more times. The other data points with small con could not be clustered correctly. Therefore, it is necessary to have roughly equal con among data points so as to get a correct clustering result of all data points.

Moreover, a subspace is grouped if any two data points in the subspace is connected by a couple of points which belong to the same subspace [20], which means a data point can be clustered correctly as long as it has a connection to the right group (one connection is enough). There are totally 2Nk connections formed in the process of OMP, while not all of them are necessary or effective. Excessive connections of one point will weaken the space detection property (SDP) [21] of the similarity matrix, which leads to a decrease of the clustering accuracy. Therefore, we consider restricting the number of connections that each data point could have.

3.2 Algorithm

During the first iteration of OMP, we restrict the number of connections of each data point, and this algorithm is named as RCOMP. In practice, we only take connection-1 (connection from xi to the first neighbor it chooses) and connection-2 (connections from xi to {xj s.t.j$\neq$i} who choose xi as the first neighbor) as connections since they have the largest coefficients and the rest is insignificant in contrast. Discarding the connections with negligible coefficients relaxes the limit on selecting neighbors and reduces the computational time. The whole procedure of RCOMP is shown in Algorithm 1.

Definition 2** (control matrix)**

Given a dataset X = [x1, x2 ,…, xN]* $\in$ $\mathbb{R}$D×N, control matrix M $\in$ $\mathbb{R}$N×N, Mij is the control coefficient which is applied to multiplying the inner products of xi (or its residuals) and xj(s.t.j$\neq$i) so as to control the neighbor selection.*

OMP selects xj that has the largest absolute inner product with xi as the first neighbor of xi, so that we can control the selection by controlling the inner product of xi and {xj s.t. j$\neq$i} (Algorithm 2). We present a N$\times$N control matrix (M) to realize RCOMP by multiplying the inner products (dot) with the corresponding entries of M as shown in (5). The number of connections of xj could still have (nconj) decreases when a new connection formed (step 5). When nconj reduces to 0, which means the number of connections of xj reaches the limit, the xj could not be chosen as the neighbor of the other data points anymore(step 8).

[TABLE]

3.3 Analysis

The sparse solution is not unique since any d**i linearly independent points from S**i can represent a point xi$\in$S**i [15]. Therefore, we can still get a sparse representation of xi even if we change the points that OMP chooses.

If there are enough candidates for the data points behind to choose as neighbors, the restriction of RCOMP would not worsen the clustering result. As a consequence, the number of connections each data point could have (rcon) is related to the size of the dataset. The larger the dataset, the smaller the rcon. In particular, when rcon is equal to 2, the RCOMP would form a linear structure among data points (Fig.2.(b)). Generally, we set rcon = 2, while rcon = 3 for small datasets in extreme situation.

We can get a roughly equal con among data points by restricting it to rcon since the total number of con is instant, which increases the number of connections of data points whose con is relatively small in OMP. Therefore, all the data points in X can be clustered correctly. As shown in Fig.2., in the case where the total number of connections of all data points is the same, RCOMP tends to own more data points whose connections is more than one (green points). And the dotted points mean they can t be connected again.

[TABLE]

Furthermore, when the subspaces are not disjoint (or there is noise), OMP may forms too many wrong connections with data points (X0 ={xj} in (6)) which are near the intersections between subspaces. In contrast, RCOMP allows only 2 (in general) connections of one data point, so that there would be at most 2 wrong connections of each data point in X0. Therefore, the restriction of RCOMP makes it robust to the noise.

Notice that OMP selects neighbors by the absolute value of inner products, which is updated by M. Therefore, most of the data point selection strategies can be implemented on the framework of control matrix M simply by updating the control coefficients of it. For instance, setting the ith column of M to be zero actually drops the ith data point just as AOMP-SSC dose.

4 EXPERIMENTS

4.1 Synthetic Experiment

We randomly generate n $=$ 3 subspaces of dimension d $=$ 6 in 40-D ambient space. There are 10 to 500 data points in every subspace. The noise rate is 0.8 in Fig.2.(a). We set up rcon $=$ 2. The average results of 100 trials are shown in Fig.2.

All of the three algorithms display good performance when there is no noise (only under this experiment settings). And RCOMP exceeds OMP at the presence of noise in Fig.2(a). More importantly, as data density grows, there are more and more data points near the intersections of subspaces. As a consequence, OMP suffers from an accuracy decline, while RCOMP keeps good performance. Therefore, we come to a conclusion that RCOMP is more robust than OMP in large and noisy datasets, which will be verified again in the following two experiments. Similar results in connectivity [22] is shown in Fig.2(b).

In order to test the performance of RCOMP under different rates of noise, we set noise rates various from 0 to 0.9 with 200 data points of each subspace. Other settings are the same as the above experiment. Fig.2 (c) shows that RCOMP outperforms OMP in various rates of noise.

Although SSC outperforms RCOMP in terms of accuracy and connectivity, RCOMP is several magnitudes faster than SSC (Fig.2(d)). This is in line with the intention that RCOMP aims at keeping a balance between accuracy and efficiency. Moreover, RCOMP achieves better performance over SSC in the next two real-world datasets.

4.2 Handwritten Digits Images Segmentation

We use Usps, a handwritten 0-9 digits database of 8-bit grayscale images, to test RCOMP in contrast to other methods. There are 1100 images for each digit, from which we randomly choose $\{$200, 400, 600, 800, 1000$\}$. All of the image are projected to dimension 200 using PCA. We set iteration k $=$ 3, rcon $=$ 2 since the dataset is large enough, and the number of clusters we choose is 5. After 100 times trials, the result of accuracy and computational time is shown in Table.1. RCOMP outperforms the others in terms of accuracy in all five groups, which shows the robustness of RCOMP in various sizes of datasets. Moreover, RCOMP is significantly faster than the other clustering methods (except for OMP), which makes it preferable for large-size tasks.

4.3 Face Image Clustering

In this experiment, we test our algorithm on the Extended Yale B dataset [23], which consists of frontal face images of 38 persons under 64 different illumination conditions. The size of each image is down sampled to 48$\times$42 pixel. We randomly pick n $\in$ $\{5$, 10, 20, 30, 38$\}$ persons with all the images of each. We set the iteration time k $=$ 5, rcon $=$ 3 since there are only 64 data points of each subspace. The result is shown in Table.2 with 100 trials of each n, which demonstrates the superiority of RCOMP in multiple classes.

5 CONCLUSIONS

We propose a noise-robust RCOMP-SSC algorithm, which obtains a roughly equal number of connections of each data point and alleviates the effects of data points near the intersections of subspaces by restricts the number of connections of each data point. And it is realized on the scalable framework of control matrix we develop. We illustrate the inspiration behind our algorithm and present theoretic analysis of it. Experiments on synthetic data and different tasks of computer vision show that our algorithm outperforms other clustering methods in terms of accuracy, computational time and scalability in various sizes and classes of datasets.