Three-Stage Subspace Clustering Framework with Graph-Based Transformation and Optimization

TL;DR

This paper introduces a three-stage subspace clustering framework that enhances affinity matrix quality using graph-based transformation and optimization, leading to more accurate clustering of high-dimensional data.

Contribution

The paper proposes a universal three-stage framework with a novel graph-based transformation and optimization step to improve subspace clustering accuracy.

Findings

Improved clustering accuracy on synthetic and real-world datasets.

Enhanced affinity matrix connectivity and authenticity.

Framework applicable to various regularization-based SC methods.

Abstract

Subspace clustering (SC) refers to the problem of clustering high-dimensional data into a union of low-dimensional subspaces. Based on spectral clustering, state-of-the-art approaches solve SC problem within a two-stage framework. In the first stage, data representation techniques are applied to draw an affinity matrix from the original data. In the second stage, spectral clustering is directly applied to the affinity matrix so that data can be grouped into different subspaces. However, the affinity matrix obtained in the first stage usually fails to reveal the authentic relationship between data points, which leads to inaccurate clustering results. In this paper, we propose a universal Three-Stage Subspace Clustering framework (3S-SC). Graph-Based Transformation and Optimization (GBTO) is added between data representation and spectral clustering. The affinity matrix is obtained in the first stage, then it goes through the second stage, where the proposed GBTO is applied to generate a reconstructed affinity matrix with more authentic similarity between data points. Spectral clustering is applied after GBTO, which is the third stage. We verify our 3S-SC framework with GBTO through theoretical analysis. Experiments on both synthetic data and the real-world data sets of handwritten digits and human faces demonstrate the universality of the proposed 3S-SC framework in improving the connectivity and accuracy of SC methods based on $\ell_0$, $\ell_1$, $\ell_2$ or nuclear norm regularization.

Equations

$$x_j=X{c}_j,c_{jj}=0,\text{mbox}orequivalentlyX=XC,diag(C)=0,$$

$$w_{ij}=\left|c_{ij}\right|+\left|c_{ji}\right|\text{mbox}orequivalentlyW=|C|+|C^T|,$$

$$d_{ij}^{\ast }=\min (d_{ij},d_{ik}+d_{kj})$$

$$\begin{array}{r}{w}_{ij}^{\ast }=\mathrm{DWT}(d_{ij}^{\ast })=\mathrm{DWT}(\min (d_{ij},d_{ik}+d_{kj}))\\ =\mathrm{DWT}(\min (\mathrm{WDT}(w_{ij}),\mathrm{WDT}(w_{ik})+\mathrm{WDT}(w_{kj})))\\ =\max (w_{ij},\mathrm{DWT}(\mathrm{WDT}(w_{ik})+\mathrm{WDT}(w_{kj})))\end{array}$$

$$c_i=\underset{c_i}{\mathrm{argmin}}{\Vert c_i\Vert }_0\text{ s.t. }{x}_i=X{c}_i,c_{ii}=0$$

$$x_i\in S_i,\forall j=1,...,N,c_{ij}\ne 0⟹x_i\in S_i.$$

$$c_{ij}=0,c_{ik}\ne 0,c_{kj}\ne 0⟹x_i\in S_k,x_j\in S_k,$$

$$c_{ij}\ll c_{ik},c_{ij}\ll c_{kj}⟹d_{ij}\gg d_{ik},d_{ij}\gg d_{kj}\text{mbox}orequivalently{d}_{ij}\gg d_{ik}+d_{kj},$$

$$\begin{array}{r}{c}_{ij}^{\ast }=\frac{1}{2}{w}_{ij}^{\ast }=\frac{1}{2}\max (w_{ij},\mathrm{DWT}(\mathrm{WDT}(w_{ij})+\mathrm{WDT}(w_{kj})))\\ =\max (\frac{1}{2}{w}_{ij},\mathrm{DWT}(\mathrm{WDT}(\frac{1}{2}{w}_{ij})+\mathrm{WDT}(\frac{1}{2}{w}_{kj})))\\ =\max (c_{ij},\mathrm{DWT}(\mathrm{WDT}(c_{ij})+\mathrm{WDT}(c_{kj})))\end{array}$$

$$Ncut(A,B)=\frac{\mathrm{cut}(A,B)}{\mathrm{assoc}(A,G)}+\frac{\mathrm{cut}(B,A)}{\mathrm{assoc}(B,G)},$$

$$Ncut(A,B)=\frac{{\sum }_{\left(e_i>0,e_j<0\right)}-w_{ij}{e}_i{e}_j}{{\sum }_{e_i>0}{d}_i}+\frac{{\sum }_{\left(e_i<0,e_j>0\right)}-w_{ij}{e}_i{e}_j}{{\sum }_{e_i<0}{d}_i},$$

$$Ncut\left(A_1,A_2,\dots A_k\right)=\frac{1}{2}\sum _{i=1}^k\frac{\mathrm{cut}\left(A_i,{\overline{A}}_i\right)}{W\left(A_i\right)}=\sum _{i=1}^k\frac{\mathrm{cut}\left(A_i,{\overline{A}}_i\right)}{\mathrm{vol}\left(A_i\right)},$$

$$\mathrm{vol}(A_i^{\ast })-\mathrm{vol}(A_i)\gg \mathrm{cut}\left(A_i^{\ast },\overline{A_i^{\ast }}\right)-\mathrm{cut}\left(A_i,{\overline{A}}_i\right).$$

$$\sum _{i=1}^k\frac{\mathrm{cut}\left(A_i^{\ast },\overline{A_i^{\ast }}\right)}{\mathrm{vol}\left(A_i^{\ast }\right)}<\sum _{i=1}^k\frac{\mathrm{cut}\left(A_i,{\overline{A}}_i\right)}{\mathrm{vol}\left(A_i\right)}$$

$$\mathrm{sparsity}(D^{\ast })\le \mathrm{sparsity}(D)⟹\mathrm{sparsity}(W^{\ast })\le \mathrm{sparsity}(W).$$

Full text

1 Introduction

High-dimensional data, such as images and documents, are ubiquitous in many applications of computer vision, e.g., face clustering [1], image representation and compression [2], and motion segmentation [3,4]. In order to deal with these data in a high-dimensional ambient space, a union of low-dimensional data is adopted to approximate the original high-dimensional data, which is known as Subspace Clustering (SC) [5]. The task of SC is to partition the data into groups so that data points in a group come from the same subspace.

Many methods for subspace clustering have been proposed, including algebraic [6], iterative [7]-[10], statistical [11]-[16], and spectral clustering based methods [17]-[27]. State-of-the-art methods tend to adopt spectral clustering due to its advantage to reveal the relationship between data points based on graph. In spectral clustering, data points are viewed as vertexes in set $X=\left[x_{1},…,x_{N}\right]\in\mathbb{R}^{D\times N}$ in an undirected graph $G=\left(X,E\right)$, and the task is to partition $G$ into different subgraphs. Usually, an affinity matrix (or similarity matrix) $W$ which has the similarity between each vertex pair as its elements is generated to represent $G$. By applying spectral clustering to $W$, the data points will be clustered into different subspaces precisely.

Spectral clustering based methods usually follow a two-stage framework. In the first stage, with data representation techniques, a coefficient matrix $C$ is generated from the original data based on the self-expressiveness property of data belonging to the same subspace, and then an affinity matrix $W$ is obtained from $C$. That is,

[TABLE]

[TABLE]

where $X=[x_{1},…,x_{N}]\in\mathbb{R}^{D\times N}$is the data matrix, $C=[c_{1},…,c_{N}]\in\mathbb{R}^{D\times N}$ is the coefficients matrix, and $W$ is the affinity matrix. Then spectral clustering is directly applied to $W$ to partition data points into subspaces in the second stage.

However, the affinity matrix $W$ obtained in the first stage of the two-stage framework contains insufficient information of $G$ since it’s usually sparse or low rank, which is obviously not capable to represent the relationship between data points and inappropriate to be applied with spectral clustering directly. Applying spectral clustering to $G$ will lead to inaccurate results especially in some data sets with large number of data points. Two factors lead to this problem. Firstly, too much $zero$ elements in $W$ deny potential relations among data points, which makes the number of connections between data points particularly small. Secondly, nonzero elements in $W$ cannot reveal the authentic similarity between data points.

In this paper, we propose a universal Three-Stage Subspace Clustering framework (3S-SC), which aims to overcome the drawbacks of the two-stage SC framework. It provides a Graph-Based Transformation and Optimization (GBTO) mechanism which turns the original $W$ obtained from data representation techniques into an optimized affinity matrix $W^{*}$ which is more capable to represent the distribution of data points in the high-dimensional ambient space. In GBTO, we adopt the classic Floyd-Warshall algorithm as an optimization strategy to solve this SC problem. Besides, depending on different application scenarios, we propose two implementations of the 3S-SC framework which are Hard 3S-SC and Soft 3S-SC.

In addition, we compare and analyze different transformation strategies in GBTO which determines how weight and distance are transformed into each other. Finally, we set experiments on synthetic data and real-world data sets of handwritten digits as well as human faces with varying lighting to verify our theoretical analysis. The universality of the 3S-SC framework in improving the accuracy and connectivity of SC methods with different norm regularizations is demonstrated in comparison with traditional two-stage framework.

Related work.

Current spectral clustering based methods tend to apply $\ell_{0}$, $\ell_{1}$, $\ell_{2}$ or nuclear norm regularization on the coefficient matrix $C$, including original Sparse Subspace Clustering using $\ell_{1}$ norm (SSC-$\ell_{1}$) [28][29], Least Squares Regression (LSR) [17] using $\ell_{2}$ norm, Low Rank Representation (LRR) [30][31] using nuclear norm, Elastic Net Subspace Clustering (ENSC) [27] using a mixture of $\ell_{1}$ and $\ell_{2}$ norm and SSC by Orthogonal Matching Pursuit (OMP) [26] using $\ell_{0}$ norm. Meanwhile, Block Diagonal Representation (BDR) [32] uses block diagonal matrix induced regularizer to directly pursue the block diagonal matrix. Such methods divide the SC problem into two steps described as two-stage SC framework. Though Structured Sparse Subspace Clustering (SSSC) [33] uses a joint affinity learning and subspace clustering framework to re-weight the representation matrix, it still ignores that $W$ generated by (SSC-$\ell_{1}$) in the first stage cannot represent the authentic distribution of high-dimensional data points. Our paper presents a more universal framework which can be applied to SC methods with different data representation techniques in the first stage.

2 Three-Stage Subspace Clustering: A universal framework

The main difference between these spectral clustering based SC methods lies in how the affinity matrix is obtained. Therefore, these algorithms can be concluded in a Two-Stage Subspace Clustering framework. In order to deal with its drawbacks mentioned in Section 1.1, we propose a Three-Stage Subspace Clustering, which is universal for state-of-the-art SC algorithms. Step 3 in Algorithm 1 is added in comparison with two-stage SC. As is shown in Figure 1, 3S-SC contains Graph-Based Transformation and Optimization, details of which is in Section 3. Depending on the application scenario, two implementations of the 3S-SC framework are proposed:

• •

Hard 3S-SC: in this case all elements in affinity matrix $W$ are applied with Graph-Based transformation and optimization, in which way the number of nonzero entries in $W$ will increase. It works well especially in data sets with plenty subspaces and data points.

• •

Soft 3S-SC: in this case only nonzero elements in W are applied with GBTO, which will not change the sparsity in $W$. It just reconstructs the existing relations by adjusting similarity.

3 Graph-based Transformation and Optimization (GBTO)

In the last stage of 3S-SC, we apply spectral clustering to get SC results. To accomplish this, an affinity (or similarity) matrix $W\in\mathbb{R}^{N\times N}$ containing sufficient information of how high-dimensional data points distribute needs to be generated. As described in Section 1, the affinity matrix $W\in\mathbb{R}^{N\times N}$ obtained in the first stage is usually sparse or low rank, which is obviously unqualified to represent the relationship between data points and inappropriate to be applied with spectral clustering directly. To solve this problem, Graph-based Transformation and Optimization algorithm (GBTO) is proposed.

Definition 1 (Simulated distances and simulated graph)

Elements in matrix $D$ generated from the affinity matrix $W$ to represent distances between data points are called simulated distances and $D$ is called simulated distances matrix, while the graph containing data points with simulated distances is defined as simulated graph.

Stage 2 in Figure 1 describes how GBTO works. By transforming the affinity matrix $W$ into simulated distances matrix $D\in\mathbb{R}^{N\times N}$ via Weight-Distance Transformation (WDT), a simulated undirected graph $G$ is generated. Then $D^{*}\in\mathbb{R}^{N\times N}$ with minimized distances is generated from $D$ by applying Floyd-Warshall, after which a new affinity matrix $W^{*}$ with optimized similarity is generated by applying Distance-Weight Transformation (DWT) to $D^{*}$.

Obviously, GBTO can be divided into two parts: transformation between weights and distances (section 3.1), and optimization for simulated distances which determines how the relationship between data points is reconstructed (section 3.2).

3.1 Transformation strategy between weights and distances (WDT and DWT)

In an undirected graph $G=(X,E)$ with vertex set $X$, the weight (or similarity) between two data points grows approximately in inverse proportion to the distance between them, and the transformation should obey this rule as well. Due to the column normalization, the weight are distributed in the range $[0,1]$. Thus WDT and DWT can be approximated as $f(x)=\tfrac{1}{x}(0<x<1)$, where $x$ here stands for weights. Both f(x) and its inverse transformation $f^{-1}(x)$ are simple in computation. As is shown in Figure 2, compared with another proposed transformation $g(x)=1-$ln$x(0<x<1)$, $f(x)$ enjoys more stable changes when $x$ approaches $zero$, which means the $f(x)$ transformation pays more attention to small weights. It’s a better mapping for GBTO aiming to deal with abnormally small similarity which is irrational in ternary relationship as described later in (6) and (7). Thus, transformation algorithms between affinity matrix and distances matrix are proposed as below.

WDT is applied to the original affinity matrix $W$, and it transforms $W$ into a simulated distances matrix. After optimization for simulated distances, $D^{*}\in\mathbb{R}^{N\times N}$ is generated from $D$ and it needs transforming back into affinity matrix so that spectral clustering can be applied. Thus the inverse transformation of WDT which is called DWT is proposed as follows.

3.2 Optimization strategy for simulated distances matrix D

After WDT, the weights are transformed into simulated distances so that graph-based optimization can be applied. $Zeros$ in $W$ means no similarity between data points, so the corresponding distance is defined as $infinity$. Elements in $D$ such as $d_{ij}$ stands for the distance from data point $x_{i}$ to $x_{j}$, and it’s obvious that $D$ is symmetric which has $zeros$ as diagonal elements. In the first stage, elements in $W$ fail to reveal the authentic relations between data points, thus the simulated distances generated from $W$ are not precise either. In the simulated undirected graph $G=(X,E)$ with vertex set $X=\left[x_{1},…,x_{N}\right]$, many potential connections are not established, which results in extremely overlarge distances or even infinite distances. So it’s a problem of how to minimize the distances between data points. We adopt the classic Floyd-Warshall algorithm [40] which aims to find the shortest paths between all pairs of vertices in a weighted graph with positive or negative edge weights, after which $D$ can be optimized as $D^{*}$ with shortest distances between all data points.

Theorem 1.

Floyd-Warshall minimizes the distances between high-dimensional data points.

[TABLE]

Corollary 1.

3S-SC with GBTO optimizes the similarity between data points.

[TABLE]

After traversal of all intermediate points $x_{k}$, $k\in$ $\{1,...,N\}$ , the Floyd–Warshall algorithm compares all potential paths through the graph $G$ between each pair of data points. The left upper of Figure 3 is the simulated graph $G=(X,E)$ as proposed in Section 3.1. After Floyd-Warshall, some new connections have been created which are shown as dotted lines in the right upper of Figure 3. Besides, some existing connections are optimized by minimizing the distances via intermediate nodes, such as the connection between node $x_{i}$ and $x_{j}$ which is optimized via intermediate node $x_{k}$.

3.3 3S-SC applied with GBTO

With the implementation of GBTO, 3S-SC can be applied to most of spectral clustering based SC methods. 3S-SC with GBTO applied with Algorithm 5 is defined as Hard 3S-SC, which means both zero and nonzero elements in affinity matrix $W$ are optimized. This will lead to more nonzero elements in affinity matrix. For Soft 3S-SC, restriction as $d_{ij}\not=\infty$ is added in step 6 of Algorithm 5 to ensure only nonzero elements in $W$ are optimized, in which way only the existing relations are optimized and there is no risk that wrong connections between subspaces are established. Compared with Hard 3S-SC, Soft 3S-SC is a more conservative optimization method which improves spectral clustering accuracy without changing sparsity.

[TABLE]

In the first stage, different techniques are applied to solve the problem in (5). Usually it’s simplified as a $\ell_{1}$ or $\ell_{2}$ problem which is not NP hard. $c_{i}$ in $C=[c_{1},…,c_{N}]$ are coefficients for $x_{i}$ to be written as a linear combination of other points based on self-expressiveness. Multiple solutions for $C$ vary from each other, so self-expressiveness could not show the global linear correlation of data.

Lemma 1.

3S-SC with GBTO optimizes the expression of linear correlation of data points.

With $S_{i}$ standing for subspace, (1) shows the self-expressiveness property of data lying in the same subspace, which can be derived that:

[TABLE]

However, the coefficients matrix $C$ obtained in the first stage doesn’t possess good self-expressiveness. For example, if we initialize $x_{i}\in S_{i},x_{k}\in S_{k},x_{j}\in S_{j}$, in some special cases:

[TABLE]

where $x_{i}$ and $x_{j}$ belong to the same subspace, while the opposite conclusion can be derived from $c_{ij}=0$. More commonly, for some nonzero entries in $C$:

[TABLE]

the distance from intermediate point $x_{k}$ to $x_{i}$ and $x_{j}$ are small while $x_{i}$ is far from $x_{j}$, which is irrational in ternary relationship. After GBTO, the distances between data points are minimized and the similarity is optimized, which can be viewed as optimization for $C$ as well. That is:

[TABLE]

Thus, 3S-SC with GBTO optimizes the data representation to possess better linear correlation expression.

Lemma 2.

In spectral clustering, higher similarity of edges inside subgraphs leads to smaller cut.

State-of-the-art spectral clustering adopt Ncut [34] to gain a more accurate result.Ncut is defined as :

[TABLE]

where $cut$ is the degree of dissimilarity between these two subgraphs $A$ and $B$, which can be computed as total weight of the edges that have been removed, and $\operatorname{assoc}(A,G)=\sum_{u\in A,t\in G}w(u,t)$is the total connection from vertexes in $A$ to all vertexes in graph $G$. It can be rewritten as:

[TABLE]

where $e_{i}=1$ if vertex $x_{i}\in A$ and $e_{i}-1$ if $x_{i}\notin A$, and $d(i)=\sum_{j}w_{ij}$. For the task of partitioning graph into $k$ pieces, the Ncut for SC can be rewritten as:

[TABLE]

where $\operatorname{vol}\left(A_{i}\right)=\sum_{j\in A_{i}}d_{j}$ is defined as the sum of the weights of all edges in subgraph $A_{i}$ whose nodes belong to $S_{i}$, and $\overline{A}_{i}$ is graph without $A_{i}$. After applied with 3S-SC, $A^{*}_{i}$ with optimized similarity is generated. The increment of weights inside subgraphs is much larger than that between subgraphs especially in graph with more subgraphs and more nodes per subgraph. That is:

[TABLE]

This is obvious since data points belong to the same subspace have larger similarity and more connections inside subgraphs are established. So it can be derived that:

[TABLE]

Theorem 2.

3S-SC with GBTO improves accuracy and connectivity by higher similarity which is closer to the truth.

[TABLE]

This can be concluded from (4) and (15), which show how 3S-SC affects the similarity. This means 3S-SC with GBTO improves clustering accuracy and connectivity by optimized similarity.

4 Experiments

We compare the performance of spectral clustering based SC methods with these implemented as 3S-SC methods, including SSC-$\ell_{1}$, OMP, ENSC, LRR, LSR and BDR, which contain $\ell_{0}$, $\ell_{1}$, $\ell_{2}$ and nuclear norm. These methods applied with hard 3S-SC are prefixed with ’3S’, such as 3S-SSC-$\ell_{1}$. Those applied with soft 3S-SC are prefixed with ’soft’. Parameters are set as recommended ($\lambda=0.9$ for ENSC, $\lambda=50,\gamma=1$ for BDR). Experiments are set on synthetic data, handwritten digits set MNIST [35] and USPS [36], and human faces set Extended Yale B (EYaleB)[37] with 50 trails. Connectivity, Clustering Accuracy and Normalized Mutual Information (NMI) [38][39] are metrics to evaluate the performance. Connectivity is defined as the second smallest eigenvalue $\lambda_{2}$ of the normalized Laplacian $L=I-D^{-1/2}WD^{-1/2}$, where $D=Diag(W\cdot 1)$ is the degree matrix of graph $G$. NMI quantifies the amount of information obtained by clustering results compared with the ground-truth.

Synthetic Experiments. Figure 4 shows how clustering accuracy changes with samples per subspace on synthetic data with $n=5$ subspaces each of dimension $d=8$ in ambient space of dimension $D=12$ ($D=18,d=7$ for BDR). Hard 3S-SC improves accuracy for LSR, LRR and BDR, while soft 3S-SC works well in SSC-$\ell_{1}$, OMP and ENSC with $\ell_{0}$ and $\ell_{1}$ norm. NMI synchronously changes with accuracy, and connectivity of 3S-SC methods is usually higher than 80%.

Clustering Handwritten Digits. Experiments on two data sets are set differently. The number of subjects chosen to be clustered changes on MNIST, while samples per subspace changes on USPS. The feature vectors for images in MNIST are projected to dimension 500 and 200 for USPS by PCA. As is shown in Table 1, Figure 5(a) and 5(b), in real-world data sets of handwritten digits, 3S-SC methods usually obtain higher accuracy and NMI. In terms of data sets with more samples per subspace, GBTO works better due to more authentic relations between data points.

Clustering Human Faces with Varying Lighting. Data points are images downsampled from $192\times 168$ to $48\times 42$. Table 2 shows accuracy and Figure 5(c) reports connectivity. For OMP, slight improvements has been made when applied with soft 3S-SC since no new connections are created. For ENSC and BDR, accuracy decrease of 3S-SC methods occurs when clustering 5 or 10 subjects because more connections between subspaces are established compared with inner connections when data sets are small. Considerable increase can be observed when clustered subjects are more than 10.

The time complexity of Floyd-Warshall is $O\left(n^{3}\right)$, while the time cost of 3S-SC is still acceptable even on limited computing resources. For larger and more complex data sets, techniques like vectorization for loops in ®MATLAB could reduce time. Moreover, the performance of BDR varies greatly when $\lambda$ and $\gamma$ are set differently, while 3S-BDR overcomes the over-dependence on parameters.

5 Conclusion

We proposed a Three-Stage Subspace Clustering framework (3S-SC) with two implementations (Hard & Soft), in which Graph-Based Transformation and Optimization (GBTO) is applied to optimize the representation of authentic data distribution. 3S-SC is universal for SC methods with different regularizations, and the effectiveness of it is demonstrated on several data sets. We note that 3S-SC sometimes doesn’t work well in small data sets and this is left for future research.

Acknowledgments

This work was supported in part by the Shenzhen Municipal Development and Reform Commission (Disciplinary Development Program for Data Science and Intelligent Computing), in part by Shenzhen International cooperative research projects GJHZ20170313150021171, and in part by NSFC-Shenzhen Robot Jointed Founding (U1613215).