Simultaneous Subspace Clustering and Cluster Number Estimating based on Triplet Relationship

TL;DR

This paper introduces a novel triplet relationship-based framework for subspace clustering that simultaneously estimates the number of clusters and improves robustness over traditional pairwise methods, especially at subspace intersections.

Contribution

The paper proposes a new hyper-correlation structure called triplet relationship, enhancing robustness and enabling automatic cluster number estimation in subspace clustering.

Findings

Outperforms existing methods on synthetic datasets.

Effective in real-world high-dimensional data.

Automatically determines the optimal number of clusters.

Abstract

In this paper we propose a unified framework to simultaneously discover the number of clusters and group the data points into them using subspace clustering. Real data distributed in a high-dimensional space can be disentangled into a union of low-dimensional subspaces, which can benefit various applications. To explore such intrinsic structure, state-of-the-art subspace clustering approaches often optimize a self-representation problem among all samples, to construct a pairwise affinity graph for spectral clustering. However, a graph with pairwise similarities lacks robustness for segmentation, especially for samples which lie on the intersection of two subspaces. To address this problem, we design a hyper-correlation based data structure termed as the \textit{triplet relationship}, which reveals high relevance and local compactness among three samples. The triplet relationship can be derived from the self-representation matrix, and be utilized to iteratively assign the data points to clusters. Three samples in each triplet are encouraged to be highly correlated and are considered as a meta-element during clustering, which show more robustness than pairwise relationships when segmenting two densely distributed subspaces. Based on the triplet relationship, we propose a unified optimizing scheme to automatically calculate clustering assignments. Specifically, we optimize a model selection reward and a fusion reward by simultaneously maximizing the similarity of triplets from different clusters while minimizing the correlation of triplets from same cluster. The proposed algorithm also automatically reveals the number of clusters and fuses groups to avoid over-segmentation. Extensive experimental results on both synthetic and real-world datasets validate the effectiveness and robustness of the proposed method.

Tables

Table 1. TABLE I: Summary of the main notations in manuscript and the corresponding descriptions.

Notation	Description
$𝒙,𝑿$	data sample, a set of samples
${𝑿}_{-j}$	samples in $𝑿$ without ${𝒙}_j$
$S$	a subspace in the ambient space
$𝝉,𝑻$	triplet, a set of triplets
$d,D$	dimensionality of subspace, dimensionality of sample
$n,N$	number of triplets, number of samples
$K,\tilde{K},\hat{K}$	real number of clusters, initialized number of cluster centers, estimated number of clusters
$𝑪$	similarity matrix derived from subspace representation method
${𝑪}^{\ast }$	binary similarity matrix preserving the top $m$ values in each row of $𝑪$ and modifying them to $1$
$c_{ij}$	$ij$-th entry of the similarity matrix $𝑪$
${𝒄}_i$	$i$-th column of $𝑪$
${𝑵}_m\left(𝒙\right)$	set of $m$ nearest neighbors of $𝒙$
$𝒞$	cluster center which contains part of samples in a subspace
${𝑻}_{in}^I,{𝑻}_{out}^I$	set of triplets which are already/not assigned into clusters in the $I$-th iteration
${𝑿}_{in}^I,{𝑿}_{out}^I$	set of samples which are already/not assigned into clusters in the $I$-th iteration, preserving the frequency
$R_m\left(𝒞\right)$	model selection reward of $𝒞$
$R_f^i\left({𝒞}_i|𝒙\right)$	fusion reward that $𝒙$ being fused into ${𝒞}_i$
$s_{ij}$	connection score of ${𝒙}_i$ toward ${𝒙}_j$
$\rho (𝝉,𝑿)$	local density of $𝝉$ against $𝑿$
${𝒢}_i,𝓖$	one of the result groups, set of the result groups
NCe	deviation rate between the estimated $\hat{K}$ and $K$
$𝒜$	error rate of the triplets

Table 2. TABLE II: Overall Comparison among comparative methods on subsets of the extended Yale B and COIL-20 datasets. The similarity matrix 𝑪 𝑪 \bm{C} calculated by both SMR and LSR is utilized as the correlation matrix of SCAMS, DP, SVD and the proposed autoSC. The best results are in bold font while ∗ indicates the second best performance.

$𝑪$	Clustering	Metrics	extended Yale B					COIL-20
			8	15	25	30	38	5	10	15	20
LSR [46]	SCAMS [13, 29]	NCe	5.21	14.00	17.12	21.25	23.00	4.36	9.00	18.32	21.00
		NMI	0.1652	0.0643	0.1544	0.3821	0.4236	0.2435	0.1524	0.1124	0.1728
SMR [32]	SCAMS [13, 29]	NCe	9.26	23.60	41.39	76.22	81.00	8.48	19.72	32.40	37.00
		NMI	0.7183	0.7272	0.6992	0.7266	0.7425	0.5885	0.6527	0.6668	0.6712
LSR [46]	DP [19]	NCe	7.90	98.38	127.92	308.00	341.00	10.90	14.70	301.05	228.00
		NMI	0.7060	0.6067	0.6245	0.6516	0.6611	0.7060	0.4984	0.6516	0.5283
SMR [32]	DP [19]	NCe	3.06	7.84	14.62	24.76	29.00	2.22	5.30	9.72	11.00
		NMI	0.6196	0.5026	0.4391	0.2166	0.2384	0.6864	0.4467	0.3643	0.3547
LSR [46]	SVD [58]	NCe	7.00	9.42	21.04	41.23	44.00	2.76	9.00	12.05	14.00
		NMI	0.2412	0.4304	0.5567	0.6523	0.6726	0.6210	0.1302	0.4092	0.4125
SMR [32]	SVD [58]	NCe	2.40	9.06	11.65	24.00	28.00	0.48	2.58	8.36	12.00
		NMI	0.7078	0.4993	0.3739	0.2808	0.2766	0.7024	0.7127	0.7224	0.7035
-	DP-space [12]	NCe	2.08	8.96	15.75	23.92	26.00	0.78	4.78	9.38	14.00
		NMI	0.0343	0.0226	0.0432	0.0406	0.0525	0.0904	0.0829	0.0718	0.0834
-	autoSC-N	NCe	0.87∗	3.16∗	4.32∗	7.68∗	9.00∗	0.75∗	2.21∗	2.42	5.00
		NMI	0.8306∗	0.7328	0.7165∗	0.6566∗	0.6871∗	0.7933∗	0.6216∗	0.7895∗	0.7126
LSR [46]	autoSC	NCe	1.08	3.32	5.79	10.50	12.00	1.50	3.40	2.00∗	4.00∗
		NMI	0.8251	0.7375∗	0.6871	0.5972	0.5833	0.7786	0.5581	0.8670	0.8239
SMR [32]	autoSC	NCe	0.76	2.08	3.15	4.98	4.00	0.38	1.18	0.80	2.00
		NMI	0.9062	0.8589	0.8432	0.8287	0.7943	0.8315	0.7701	0.7266	0.7568∗

Table 3. TABLE III: Clustering performance of the proposed autoSC with different m 𝑚 m on both extended Yale B (EYaleB) and COIL-20 (Coil-20) datasets with 15 subjects. The subspace representation is derived from SMR. Based on the results, we set m = 8 𝑚 8 m=8 in the rest of the paper.

$m$	Metrics	5	6	7	8	9	10	11
EYaleB	NCe	4.52	3.68	3.13	2.08	2.12	2.29	2.18
	NMI	0.7125	0.7736	0.8047	0.858	0.8551	0.8423	0.8536
Coil-20	NCe	1.68	1.29	0.92	0.80	0.88	0.76	0.98
	NMI	0.5647	0.6157	0.6774	0.7266	0.7107	0.7211	0.7120

Table 4. TABLE IV: Error rates of triplets ( 𝒜 𝒜 \mathcal{A} ) for the proposed autoSC on subsets of the extended Yale B (eYaleB) and COIL-20 datasets. For each row, we utilize the similarity matrix 𝑪 𝑪 \bm{C} derived from one self-representation method in the first column. The autoSC achieves consistency on the calculation of the triplets.

$𝑪$	extended Yale B					COIL-20
	8	15	25	30	38	5	10	15	20
LRR [39]	0.0155	0.0147	0.0158	0.0176	0.0169	0.0185	0.0252	0.0224	0.0231
CASS [52]	0.0158	0.0148	0.0140	0.0157	0.0162	0.0195	0.0198	0.0203	0.0193
LSR [46]	0.0144	0.0148	0.0162	0.0181	0.0172	0.0188	0.0188	0.0212	0.0199
SMR [32]	0.0135	0.0149	0.0154	0.0181	0.0161	0.0175	0.0182	0.0196	0.0202
ORGEN [33]	0.0166	0.0145	0.0151	0.0177	0.0169	0.0196	0.0210	0.0215	0.0220

Table 5. TABLE V: Comparison of time-consumption (in seconds) on the extended Yale B dataset. The best results are in bold font while ∗ indicates the second best performance.

Subjects	SCAMS	DP	SVD	DP-space	autoSC-N	autoSC
8	12.45	6.92	17.32	9.52	1.69	2.02∗
15	30.12	13.04	44.13	18.05	4.72	5.79∗
25	125.66	33.76	146.78	36.94	10.28	19.81∗
30	175.80	59.02	225.93	67.89	16.33	36.06∗
38	267.07	97.69	314.28	104.50	29.45	62.35∗

Table 6. TABLE VI: Motion segmentation performance on Hopkins 155 dataset. The row of ‘Time’ reports the average time-consumption on handling each video sequence. The best results are in bold font while ∗ indicates the second best performance. The proposed autoSC outperforms other comparative methods regarding both the estimation of K 𝐾 K and clustering.

Metrics	SCAMS	DP	SVD	DP-space	autoSC-N	autoSC
${\text{NC}}_e$	3.67	2.12	1.29	2.97	0.52∗	0.18
NMI	0.7892	0.8233	0.8670	0.7921	0.9155∗	0.9871
Time$/s$	2.68	1.22	2.45	1.56	0.26	0.32∗

Equations

$$\underset{\mathbf{C}}{\min }L(\mathbf{X}\mathbf{C},\mathbf{X})+\lambda \Vert \mathbf{C}{\Vert }_{\xi },$$

$${\mathbf{x}}_j={\mathbf{X}}_{-j}\times [c_{1j},c_{2j},\cdots ,c_{(j-1)j},c_{(j+1)j},\cdots ,c_{Nj}]$$

$${\mathbf{N}}_m({\mathbf{x}}_j)=\arg \underset{\{{\mathbf{x}}_{{\mathbf{i}}_l}\}}{\max }\sum _{l=1}^m|c_{{\mathbf{i}}_l,j}|.$$

$$1_{{\mathbf{x}}_i\in {\mathbf{N}}_m({\mathbf{x}}_j)}\times 1_{{\mathbf{x}}_j\in {\mathbf{N}}_m({\mathbf{x}}_k)}\times 1_{{\mathbf{x}}_k\in {\mathbf{N}}_m({\mathbf{x}}_i)}=1,$$

$$R_m(\mathcal{C})=\sum _i\sigma ({\mathcal{C}}_i|{\mathbf{X}}_{out}^I)-{\lambda }_m\sum _i\sigma ({\mathcal{C}}_i|{\mathbf{X}}_{in}^I),$$

$$\begin{array}{r}{R}_f^i({\mathcal{C}}_i|{\mathbf{x}}_j\in {\mathbf{X}}_{out})=\sigma ({\mathbf{x}}_j|{\mathcal{C}}_i)+{\lambda }_f\sigma ({\mathbf{N}}_m({\mathbf{x}}_j)|{\mathbf{N}}_m({\mathcal{C}}_i)),\end{array}$$

$$\rho (\mathbf{\tau },{\mathbf{X}}_{out})=\sum _{j=1}^{|\mathbf{n}|}\sigma ({\mathbf{x}}_{{\mathbf{n}}_j}|{\mathbf{X}}_{out}),$$

$$s(\bm{x}_{i},\bm{x}_{j})=f\left(\bm{x}_{i}\bigg{|}\displaystyle\bigcap_{k=1}^{n^{\prime}}\left(\bm{1}_{\bm{x}_{j}\in\bm{\tau}_{k}}\times\bm{\tau}_{k}\right)\right),$$

$$\begin{array}{rl}& \underset{\mathcal{G},\hat{K}}{\max }\sum _{k=1}^{\hat{K}}{R}_m({\mathcal{G}}_k)+\lambda \sum _{k=1}^{\hat{K}}{R}_f({\mathcal{G}}_k|\mathbf{X}),\\ & s.t.{\mathcal{G}}_k\bigcap {\mathcal{G}}_{k^{\prime }\ne k}=\varnothing ,\bigcup _{k=1}^{\hat{K}}{\mathcal{G}}_k=[1,\cdots ,N],\end{array}$$

$$\begin{array}{rl}& \mathbf{\tau }=\{{\mathbf{x}}_{{\mathbf{n}}_1},{\mathbf{x}}_{{\mathbf{n}}_2},{\mathbf{x}}_{{\mathbf{n}}_3}\}\in \mathbf{T},\\ & \text{if and only if}:c_{{\mathbf{n}}_1{\mathbf{n}}_2}^{\ast }\times c_{{\mathbf{n}}_2{\mathbf{n}}_3}^{\ast }\times c_{{\mathbf{n}}_3{\mathbf{n}}_1}^{\ast }=1,\end{array}$$

$${\mathbf{\tau }}_{ini}^I=\arg \underset{\mathbf{\tau }}{\max }\rho (\mathbf{\tau },{\mathbf{X}}_{out}^I),$$

$$\begin{array}{rl}& {\mathbf{\tau }}^{\ast }=\arg \underset{\mathbf{\tau }}{\max }\sum _{j=1}^3\sum _{\kappa }^{|\mathbf{m}|}{s}_{{\mathbf{n}}_j{\mathbf{m}}_{\kappa }},\\ & s.t.\sum _{j=1}^3\sum _{\kappa }^{|\mathbf{m}|}{s}_{{\mathbf{n}}_j{\mathbf{m}}_{\kappa }}>1;\{{\mathbf{x}}_{{\mathbf{n}}_j}{\}}_{j=1}^3\in \mathbf{\tau };\{{\mathbf{x}}_{{\mathbf{m}}_{\kappa }}{\}}_{\kappa =1}^{|\mathbf{m}|}\in \mathcal{C},\end{array}$$

$$\rho ({\mathbf{\tau }}_{ini}^{\tilde{K}+1},{\mathbf{X}}_{out}^{\tilde{K}+1})\le \rho ({\mathbf{\tau }}_{ini}^{\tilde{K}+1},{\mathbf{X}}_{in}^{\tilde{K}+1}).$$

$$s_{ij}>\min (|{\mathcal{C}}_i|,|{\mathcal{C}}_j|),$$

$${\mathcal{C}}^{\ast }=\arg \underset{{\mathcal{C}}_i}{\max }{R}_f({\mathcal{C}}_i|{\mathbf{x}}_j),i\in \{1,2,\cdots ,\hat{K}\},$$

$$s_{jk}=\Vert (S_j^I{)}^{\top }{\mathbf{x}}_k{\Vert }_F^2,$$

$${\mathcal{S}}_j^{I+1}\leftarrow {\mathcal{S}}_j^I\cup \arg \underset{{\mathbf{x}}_k\in {\mathbf{X}}_{-\mathcal{N}}}{\max }{s}_{jk}.$$

$$\mathcal{A}=\frac{1}{n}\sum _{i=1}^n\frac{3-\sigma ({\mathbf{\tau }}_i|{\mathbf{g}}_i^{\ast })}{2},$$

$${\text{NC}}_e=\frac{1}{M}\sum _{i=1}^M|{\hat{K}}_i-K|,$$

Full text

I Introduction

With the ability to disentangle latent structure of data in an unsupervised manner [2, 3, 4], subspace clustering is regarded as an important technique in the data mining community and for various computer vision applications [5, 6, 7, 8]. Traditional subspace clustering methods approximate a set of high-dimensional data samples into a union of lower-dimensional linear subspaces [9], where each subspace usually contains a subset of the samples.

In recent years, spectral clustering based methods have achieved state-of-the-art performance, taking a two-step framework as follows. First, by optimizing a self-representation problem [4, 10], a similarity matrix (and also a similarity graph) is constructed to depict the relationship (or connection) among samples. Second, spectral clustering [11] is employed for calculating the final assignment based on eigen-decomposition of the affinity graph. Note that in practice, both the number of subspaces and their dimensionalities are always unknown [4, 12]. Hence, the goals of subspace clustering include finding the appropriate number of clusters and grouping data points to them [13, 14].

Nevertheless, it is challenging to estimate the number of clusters in a unified optimization framework, since the definition of clusters is subjective, especially in the high-dimensional ambient space [13]. Also, for samples which are of different clusters but near the intersection of two subspaces, they may be even closer than samples from same cluster. This may lead to a wrong estimation with several redundant clusters, namely over-segmentation problem. Therefore, most of the spectral based subspace clustering algorithms depend on a manually given and fixed number of clusters, which cannot be generalized for multiple applications [4].

Most clustering schemes group similar patterns into the same cluster by jointly minimizing the inter-cluster similarity and the intra-cluster dissimilarity [15]. Considering the complexity of the high-dimensional ambient data space, an effective way for estimating the number of clusters is to first map the raw samples into an intrinsic correlation space, namely similarity matrix, followed by an iterative optimization according to the local and global similarity relationships derived from the projection. Elhamifar et al. [16] propose that the permuted similarity matrix can be block-diagonal, where the number of blocks is identical to the number of clusters. Moreover, Peng et al. [3] verify the intra-subspace projection dominance (IPD) of such a similarity matrix, which can be applied to self-representation optimizations with various kinds of regularizations. The IPD theory says that for two arbitrary samples from the same subspace and one from another, the generated similarities between the former samples are always larger than between latter ones in a noise-free system.

Accordingly, considering an affinity graph derived from the similarity matrix [17, 18], where the vertices denote data samples and the edge weights denote similarities, an automatic sub-graph segmentation can be greedily conducted via the following two steps inspired by the density based algorithms [19]:

1. constructing a proper number of initialized cluster centers by minimizing the weighted sum of all inter-cluster connections and maximizing the intra-cluster ones;
2. merging the remaining samples to an existing cluster by maximizing the weighted connections between the sample and the cluster.

Yet, there are also difficulties for these greedy iterative schemes. Since the ambient space can also be very dense [20], any two points which are close by evaluating the pairwise distance may not belong to the same subspace, especially for samples near the intersection of two subspaces. Consequently, a hypergraph where each edge can be connected to more than two samples [21, 22] is proposed to solve the problem in traditional pairwise graphs. In this paper, we further introduce a novel data structure termed as the triplet relationship, to explore the local geometry structure in projected space with hyper-correlations. Each triplet consists of three points and their correlations, which are considered as a meta-element for clustering. We require that all correlations are large enough, which indicates that the three points are strongly connected according to the IPD property [3].

In contrast to evaluating similarities using pairwise distances, the proposed triplet relationship demonstrates favorable performance due to the following two reasons. On one hand, it is more robust when partitioning the samples near the intersection of two subspaces since the mutual relevance among multiple samples can provide complementary information when calculating the local segmentation. On the other hand, the triplet evokes a hyper-similarity by efficiently counting the frequency of intra-triplet samples, which enables a greedy way to calculate the assignments.

Based on the newly defined triplet, in this paper, we further propose a unified framework termed as the autoSC to jointly estimate the number of clusters and group the samples by exploring the local density derived from triplet relationships. Specifically, we first calculate the self-representation for each sample via an off-the-shelf optimization scheme, followed by extracting the triplet relationship for all samples. Then, we greedily initialize a proper number of clusters via optimizing a new model selection reward, which is achieved by maximizing inter-cluster dissimilarity among triplets. Finally, we merge each of the remaining samples into an existing cluster to maximize the intra-cluster similarity by optimizing a new fusion reward. We also fuse groups to avoid over-segmentation.

The main contributions of this paper are summarized as follows:

• •

First, we define a hyper-dimensional triplet relationship which ensures a high relevance and density among three samples to reflect their local similarity. We also validate the effectiveness of triplets and distinguish them against the standard pairwise relation.

• •

Second, we design a unified framework, i.e., autoSC, based on the intrinsic geometrical structures depicted by our triplet relationships. The proposed autoSC can be used for simultaneous estimating the number of clusters and subspace clustering in a greedy way.

Extensive experiments on benchmark datasets indicate that our autoSC outperforms the state-of-the-art methods in both effectiveness and efficiency.

This paper is an extended version of our earlier conference paper [1], to which we enrich the contributions in the following five aspects: (1) We add detailed analysis of the proposed algorithm to distinguish it from comparative methods, for example, we add analysis and experimental validation on the computational complexity. (2) We provide a visualized illustration of the proposed autoSC for clearer presentation. (3) We propose a relaxation termed as the neighboring based autoSC (autoSC-N), which directly calculates the neighborhood relationship from raw data space and is more efficient than autoSC. (4) We conduct experiments on evaluating the influence of the parameter $m$ (number of preserved neighbors for each sample). (5) We experimentally evaluate our method on real-world application, i.e., motion segmentation, which also demonstrates the benefits of the proposed method.

II Related Work

Automatically approximating samples in high-dimensional ambient space by a union of low-dimensional linear subspaces is considered to be a crucial task in computer vision [9, 23, 24, 25, 26]. In this section, we review the related contributions in the following three aspects, i.e., self-representation calculation, estimating the number of clusters and hyper-graph clustering.

II-A Calculating Self-Representation

To separate a collection of data samples which are drawn from a high-dimensional space according to the latent low-dimensional structure, traditional self-expressiveness based subspace clustering method calculates a linear representation for each sample using the remaining samples as a basis set or a dictionary [27, 28]. Subspace clustering assumes that the set of data samples are drawn from a union of multiple subspaces, which can best fit the ambient space [16]. There are numerous real applications satisfying this assumption with varying degrees of exactness [29], e.g., face recognition, motion segmentation, etc.

By solving an optimization problem with self-representation loss and regularizations, subspace clustering [30, 31] calculates a similarity matrix where each entry indicates the relevance between two samples. Different regularizing schemes with various norms of the similarity matrix, e.g., $\ell_{1}$ [9], $\ell_{2}$ [32], elastic net [33] or nuclear norm [34], can explore different intrinsic properties of the neighborhood space. There are mainly three types of the regularization terms, including sparse-oriented, densely-connected and mixed norms.

Algorithms based on sparse-type norms [10, 35], e.g., $\ell_{0}$ and $\ell_{1}$ norms, eliminate most of the non-zero values in the similarity matrix to ensure that there are no connections between samples from different clusters. Elhamifar and Vidal [9] propose the sparse representation based on $\ell_{1}$ norm optimization. The obtained similarity matrix recovers a sparse subspace representation but may not satisfy the graph connectivity if the dimension of the subspace is greater than three [17]. In addition, the $\ell_{0}$ based subspace clustering methods aim to compute a sparse and subspace-preserving representation for each data sample. Yang et al. [10] present a sparse clustering method with a regularizer based on the $\ell_{0}$ norm by using the proximal gradient descent method. Numerous alternative methods have been proposed for $\ell_{0}$ minimization while avoiding non-convex problems, e.g., orthogonal matching pursuit [36] and nearest subspace neighbor [37]. The scalable sparse subspace clustering by orthogonal matching pursuit (SSC-OMP) method [27] compares elements in each column of the dot product matrix to determine which positions of the similarity matrix should be non-zero. However, this general pairwise relationship does not reflect the sample correlation well, especially for data pairs in the intersection of two subspaces [38].

In contrast, dense connection based methods, such as smooth representation [32] with $\ell_{2}$ norm and low rank representation with nuclear norm based methods [39, 40], propose to preserve many non-zero values in the similarity matrix to ensure the connectivity among intra-cluster samples [41, 42, 43]. For these densely connected frameworks [44, 45], the similarity matrix is interpreted as a projected representation of raw samples. Each column of the matrix is considered as the self-representation of a sample, and should be dense for mapping invariance (also termed as the grouping effect [46, 32]). Low-rank clustering methods [47, 48] solve a nuclear norm based optimization problem with the aim of generating a block diagonal solution with dense connections. However, the nuclear norm does not enforce subset selection well when noise exists, and the self-representation is too dense to be an efficient feature.

Neither a sparse nor dense similarity matrix reveals a comprehensive correlation structure among samples due to their conflicted nature [49, 50, 51]. Consequently, to achieve trade-off between sparsity and the grouping effect, numerous mixed norms, e.g., trace Lasso [52] and elastic net [33], have been integrated into the optimization function. Nevertheless, the structure of the data correlations depends on the data matrix, and the mixed norm is not effective for structure selection. Therefore, this method does not perform consistently well on different applications.

Recently, many frameworks that incorporate various constraints into the optimization function have been proposed to detect different intrinsic properties of the subspace [34, 53, 54]. For instance, to handle sequential samples, Guo et al. [55] explore the neighboring relationship by incorporating a new penalty, i.e., a lower triangular matrix with $-1$ on the diagonal and $1$ on the second diagonal, to force consecutive columns in the similarity matrix to be closer. In this paper, based on the intrinsic neighboring relevance and geometrical structures depicted in the similarity matrix, we calculate triplet relationships to form a hyper-correlation constraint of the clustering system. We validate the robustness of the proposed triplet relationship on top of different similarity matrices with various intrinsic properties.

II-B Estimating the Number of Clusters

Most of the real applications in computer vision require estimating the number of clusters, according to the latent distribution of data samples [9]. To solve this problem, three main techniques exists: singular-based Laplacian matrix decomposition, density-based greedy assignment and hyper-graph based segmentation.

Singular-based Laplacian matrix decomposition is common in subspace clustering [56] and spectral clustering [57] due to the availability of the similarity matrix. Liu et al. [58] propose a heuristic estimator inspired by the block-diagonal structure of the similarity matrix [42]. Specifically, they estimate the number of clusters by counting the small singular values of a normalized Laplacian matrix which should be smaller than a given cut-off threshold. These singular based methods [59, 9] are dependent on a large gap between singular values, which is limited to applications in which the subspaces are sparsely distributed in the ambient space. Meanwhile, the matrix decomposition process is time-consuming when extended to large scale problems. Recently, Li et al. propose SCAMS [60, 29] which estimates the number of clusters by minimizing the rank of a binary relationship matrix encoding the pairwise relevance among all data samples. Simultaneously, they incorporate a penalty term on the clustering cost by minimizing the Frobenius inner product of the similarity matrix and binary relationship matrix.

Density based methods [61] greedily discover both the optimal number of clusters and the assignments of data to the clusters according to the local and global densities which are calculated by the pairwise distances in ambient space. Rodriguez et al. [19] automatically cluster samples based on the assumption that each cluster center is characterized by a higher density in the weight space than all its neighbors, while different centers should be far apart enough to avoid redundancy. Specifically, for each sample, its Euclidean-based local density and the distance to any points with higher densities are iteratively calculated and updated. In each iteration, the algorithm finds a trade-off between the density of cluster centers and the inter-cluster distance to update the assignments. Wang et al. [12] employ the Bayesian nonparametric method based on a Dirichlet process, and propose DP-space, which exploits a trade-off between data fitness and model complexity. DP-space is more tolerate to noisy and outlier values than the alternative algebraic and geometric solutions. Recently, correlation clustering (CC) [62] first constructs an undirected graph with positive and negative edge weights, followed by minimizing the sum of cut weights during the segmenting process. Sequentially, the clustering assignments can be optimized with a greedy scheme. Nevertheless, most of these density based algorithms are limited to pairwise correlation when evaluating the similarity of data samples, which is not robust for densely distributed subspaces.

II-C Hyper-graph Clustering

To tackle the limitations of the pairwise relation based methods, the hyper-graph relation [63, 64, 65] is proposed and the related literature follows two different directions. Some transform the hyper-correlation into a simpler pairwise graph [21, 66], followed by a standard graph clustering method, e.g., normalized cut [11], to calculate the assignments. Besides, other methods [39, 13] explore a generalized way of extending the pairwise graph to the hyper-graph or hyper-dimensional tensor analysis. For instance, Li et al. propose a tensor affinity variant of SCAMS, i.e., SCAMSTA [13], which exploits the higher order mathematical structures by providing multiple groups of nodes in the binary matrix derived from an outer product operation on multiple indicator vectors. However, estimating the number of clusters from the rank of the affinity matrix only works well for the ideal case, and can hardly be extended to complex applications since the noise can have a significant impact on the rank of affinity matrix.

In this paper, we estimate the number of clusters by initializing the cluster centers with maximum inter-cluster dissimilarities and also maximum local densities. We calculate the initialization according to the local correlations reflected by the proposed triplet relationships, where each of them depicts a hyper-dimensional similarity among three samples and easily-evaluated relevances to other triplets. Both theoretical analysis on triplets as well as the experimental results demonstrate the effectiveness of the proposed method.

III Methodology

III-A Preliminary

The main notations in the manuscript and the corresponding descriptions are shown in Table I. Given a set of $N$ data samples $\bm{X}=\{\bm{x}_{i}\in\mathbb{R}^{D}\}_{i=1}^{N}$ lying in $K$ subspaces $\{S_{i}\}_{i=1}^{K}$ where $D$ denotes the dimensionality of each sample, spectral based subspace clustering usually takes a two-step approach to calculate the clustering assignment. First, it learns a self-representation for each sample to disentangle the subspace structure. The algorithm then employs spectral clustering [11] on the learned similarity graph derived from $\bm{C}$ for final assignments. Note that in practice, both the number of subspaces $K$ and their dimensions $\{d_{j}\}_{j=1}^{K}$ are always unknown [4, 12]. Hence, the goals of subspace clustering include finding the appropriate $K$ and assigning data points into $K$ clusters [16, 13].

In this paper, inspired by the block-diagonal structure of the similarity matrix [67], we propose to simultaneously estimate the number of clusters and assign samples into each cluster in a greedy manner. We design a novel meta-sample that we call a triplet relationship, followed by optimizing both a model selection reward and a fusion reward for clustering.

III-B Learning the Self-Representation

To explore the neighboring relationship in the ambient space $\bm{X}=\{\bm{x}_{i}\in\mathbb{R}^{D}\}_{i=1}^{N}$, typical subspace clustering methods first optimize a linear representation of each data sample using the remaining dataset as a dictionary. Specifically, spectral based subspace clustering calculates a similarity matrix $\bm{C}\in\mathbb{R}^{N\times N}$ by solving a self-representation optimization problem as follows:

[TABLE]

where $L(\cdot,\cdot):\mathbb{R}^{N\times D}-\mathbb{R}^{N\times D}\to\mathbb{R}^{+}$ denotes the reconstruction loss, $\lambda$ is the trade-off parameter and $\lVert{\cdot}\rVert_{\xi}$ denotes the regularization term where different $\xi$’s lead to various norms [4, 10], e.g., $\lVert{\bm{C}}\rVert_{1}$ [9, 16], $\lVert{\bm{C}}\rVert_{*}$ [58, 42], $\lVert{\bm{C}}\rVert_{F}^{2}$ [46], or many kinds of mixed norms like trace Lasso [52] or elastic net [33].

The $\bm{C}$ in (1) can be explained as a new representation of $\bm{X}$, where each sample $\bm{x}_{i}\in\mathbb{R}^{D}$ is mapped to $\bm{c}_{i}\in\mathbb{R}^{N}$. Furthermore, $\bm{C}$ is a pairwise distance matrix where each entry $c_{ij}$ reflects the similarity between two samples $\bm{x}_{i}$ and $\bm{x}_{j}$. Nevertheless, the pairwise distance reflects poor discriminative capacity on partitioning samples near the intersection of two subspaces. To handle this problem, in this paper, we explore a higher-dimensional similarity called Triplet relationship, which is based on a greedy combination of pairwise distances reflected by $\bm{C}$.

III-C Discovering Triplet Relationships

Given the similarity matrix $\bm{C}$ where each entry $c_{ij}$ reflects the pairwise relationship between $\bm{x}_{i}$ and $\bm{x}_{j}$, we propose to find the neighboring structure in a greedy way. For each data sample $\bm{x}_{j}\in S$, subspace clustering algorithms calculate a projected adjacent space based on the self-expressive property, i.e., each data sample can be reconstructed by a linear combination of other points in the dataset [16, 68]. Therefore, $\bm{x}_{j}$ is represented as

[TABLE]

where $\bm{X}_{-j}$ includes samples except for $\bm{x}_{j}$, which is considered as a self-expression dictionary for representation. In addition, $[c_{1j},c_{2j},\cdots,c_{(j-1)j},c_{(j+1)j},\cdots,c_{Nj}]$ records the coefficients of such combination system. With the regularization from various well-designed norms on $\bm{c}_{j}$, the optimized result of (2) is capable of preserving only linear combinations of samples in $S$ while eliminating others. Inspired by [3, 37], for each sample $\bm{x}_{j}$, we first collect its $m$ nearest neighbors, i.e. those with the top $m$ coefficients in $\bm{c}_{j}$. The $m$ nearest neighbors are defined as follows.

Definition 1**.**

*($\bm{m}$ Nearest Neighbors)  *** Let $\bm{N}_{m}(\bm{x}_{j})$ $\in$ $\mathbb{R}^{1\times m}$ denote the $m$ nearest neighbors for data point $\bm{x}_{j}$. Let:

[TABLE]

where $\bm{i}_{l}$ denotes the set of indices for the nearest neighbors, and $c_{\bm{i}_{l},j}$ denotes the coefficient between $\bm{x}_{\bm{i}_{l}}$ and $\bm{x}_{j}$.

According to Definition 1, we obtain $\bm{N}_{m}(\bm{x}_{j})$ for $\bm{x}_{j}$ which contains the $m$ samples with the $m$ largest coefficients in $\bm{c}_{j}$. The number of preserved neighbors, i.e., the parameter $m$, reflects the intrinsic dimension of the low-dimensional subspaces [69, 70], which we empirically evaluated in the experiment section. Based on the $m$ nearest neighbors, we define the triplet relationship to explore the local hyper-correlation among samples.

Definition 2**.**

*(Triplet Relationship)  *** A triplet $\bm{\tau}$ includes three samples, i.e., $\bm{\tau}=\{\bm{x}_{i},\bm{x}_{j},\bm{x}_{k}\}$, and their relationships, if and only if $\bm{x}_{i},\bm{x}_{j}$ and $\bm{x}_{k}$ satisfy:

[TABLE]

where $\bm{1}_{\bm{x}\in\bm{N}_{m}}$ denotes the indicator function which equals $1$ if $\bm{x}\in\bm{N}_{m}$ and [math] otherwise.

Based on Definition 2, we obtain $n$ triplets where we always have $n>N$, i.e., each sample is included in multiple triplet relationships. For clarity of presentation, we define a triplet matrix $\bm{T}\in\mathbb{R}^{n\times 3}$ for data samples $\bm{X}$, where each row of $\bm{T}$ records the indices of a samples in a triplet $\bm{\tau}=\{\bm{x}_{i},\bm{x}_{j},\bm{x}_{k}\}$.

Compared against the traditional pairwise relationship evoked from $\bm{C}$, the triplet incorporates complementary using the constraint in (4), which shows more robust capacity in partitioning samples near the intersection of two subspaces. Each triplet depicts a local geometrical structure which enables a better performance to estimate the density of each sample. Furthermore, the overlapped samples in multiple triplets reflect a global hyper-similarity among each other, which can be measured efficiently. Therefore, based on the triplet relationship, we can jointly estimate the number of subspaces and calculate the clustering assignment in a greedy manner.

III-D Modeling Clustering Rewards

Given $\bm{X}$, we iteratively group data samples into clusters, i.e., $\{\mathcal{C}_{i}\}_{i=1}^{\widehat{K}}$, where $\widehat{K}$ denotes the estimated number of subspaces. According to the greedy strategy, in the $I$-th iteration, the triplet $\bm{T}$ is divided into two subsets, i.e., “in-cluster” triplets $\bm{T}_{in}^{I}$ which are already assigned into clusters, and “out-of-cluster” triplets $\bm{T}_{out}^{I}$ which are still to be assigned in the subsequent iterations. For clearer presentation, we reshape both matrices $\bm{T}_{in}^{I}\in\mathbb{R}^{p\times 3}$ and $\bm{T}_{out}^{I}\in\mathbb{R}^{q\times 3}$ to vectors $\bm{X}_{in}^{I}\in\mathbb{R}^{3p}$ and $\bm{X}_{out}^{I}\in\mathbb{R}^{3q}$. In each iteration, we propose to optimize two new rewards, i.e., the model selection and the fusion reward, to simultaneously estimate the number of clusters and merge samples into respective cluster.

Definition 3**.**

*(Model Selection Reward)  *** Given $\bm{X}_{in}^{I}$ and $\bm{X}_{out}^{I}$ in the $I$-th iteration, the model selection reward $R_{m}(\mathcal{C})$ for each initialized cluster $\mathcal{C}$ in $\{\mathcal{C}_{i}\}_{i=1}^{\widehat{K}}$ is defined as:

[TABLE]

where $\sigma(\mathcal{C}|\bm{X})$ is a counting function on the frequency that $\bm{x}\in\bm{X}$ for all $\bm{x}\in\mathcal{C}_{i}$, $\lambda_{m}$ denotes the trade-off parameter.

By maximizing the model selection reward $R_{m}(\mathcal{C})$, we generate the initialized cluster $\{\mathcal{C}_{i}\}_{i=1}^{\widehat{K}}$ which has the following two advantages, where $\widehat{K}$ is the estimated number of clusters (see Fig. 1 for visualization). Firstly, the local density of sample $\bm{x}\in\mathcal{C}$ is high, i.e., $\bm{x}$ has a large amount of correlated samples in $\bm{X}_{out}$, which enables many to be merged in the next iteration. Secondly, each $\mathcal{C}$ has little correlation with samples in $\bm{X}_{in}$, which eliminates the overlap of any inter-clusters. Consequently, we can simultaneously estimate $\widehat{K}$ and initialize the clusters by optimizing the model selection reward $R_{m}$.

Definition 4**.**

*(Fusion Reward)  *** Given the initialized clusters $\{\mathcal{C}_{i}\}_{i=1}^{\widehat{K}}$, the fusion reward is defined as the probability that $\bm{x}_{j}\in\bm{X}_{out}$ is assigned into $\mathcal{C}_{i}$:

[TABLE]

where $\bm{N}_{m}(\bm{x}_{j})$ denotes the $m$ nearest neighbors of $\bm{x}_{j}$ and $\bm{N}_{m}(\mathcal{C}_{i})$ denotes the set of $m$ nearest neighbors of samples in $\mathcal{C}_{i}$, $\lambda_{f}$ denotes the trade-off parameters.

In the optimization procedure, we calculate $\widehat{K}$ fusion rewards $\{R_{f}^{i}\}_{i=1}^{\widehat{K}}$ for each $\bm{x}_{j}$, which represent the probabilities that $\bm{x}_{j}$ is assigned into clusters $\{\mathcal{C}_{i}\}_{i=1}^{\widehat{K}}$, respectively. We then merge $\bm{x}_{j}$ into the cluster with the largest fusion reward, and move $\bm{x}_{j}$ from $\bm{X}_{out}$ to $\bm{X}_{in}$.

III-E Automatic Subspace Clustering Algorithm

The first triplet for initializing a new cluster is chosen to have maximal local density. The local density $\rho$ is defined as follows.

Definition 5**.**

*(Local Density)  *** The local density $\rho$ of the triplet $\bm{\tau}$ regarding to the $\bm{X}_{out}$ is defined as follows:

[TABLE]

where $\bm{x}_{\bm{n}_{j}}$ denotes the sample in the current triplet $\bm{\tau}$ and $\bm{n}$ is the set of their indexes, $|\bm{n}|$ denotes the scale of $\bm{n}$.

Also, to measure the hyper-similarity between samples and determine the optimal triplet to merge into the initialized clusters, we define the connection score $s$ as follows.

Definition 6**.**

*(Connection Score)  *** The connection score $s$ between samples $\bm{x}_{i}$ and $\bm{x}_{j}$ is defined as:

[TABLE]

where $\bm{1}_{\bm{x}_{j}\in\bm{\tau}_{k}}$ is equal to $1$ when $\bm{x}_{j}\in\bm{\tau}_{k}$ and [math] otherwise, $n^{\prime}$ is the number of all triplets in $\bm{T}_{out}$.

We greedily optimize the proposed model selection reward $R_{m}$ and fusion reward $R_{f}$ in autoSC to simultaneously estimate the number of clusters and generate the segmentation among samples:

[TABLE]

where $\bm{\mathcal{G}}=\{\mathcal{G}_{1},\cdots,\mathcal{G}_{\widehat{K}}\}$ denotes the set of the result groups, $\widehat{K}$ is the estimated number of clusters and $[1,\cdots,N]$ denotes the universal ordinal set of samples.

We present the proposed autoSC in Fig. 1 and Algorithm 1. Specifically, the optimization includes three steps:

1. generating the triplet relationships $\bm{T}$ from the similarity matrix $\bm{C}$;
2. estimating the number of clusters $\widehat{K}$ and initializing the clusters $\mathcal{C}$;
3. assigning the samples $\bm{x}\in\bm{X}_{out}$ into proper cluster.

Calculating Triplets: The similarity matrix $\bm{C}$ reflects the correlations among samples [9], where larger values demonstrate stronger belief for the correlation between samples. For instance, $c_{ij}>c_{ik}$ indicates a larger probability for $\bm{x}_{i}$ and $\bm{x}_{j}$ being in the same cluster over $\bm{x}_{i}$ and $\bm{x}_{k}$. Accordingly, we explore the intrinsic local correlations among samples by the proposed triplets derived from $\bm{C}$.

Many subspace representations guarantee the mapping invariance via a dense similarity matrix $\bm{C}$. However, the generation of triplets relies only on the strongest connections to avoid the wrong assignment. Therefore, for each column of $\bm{C}$, i.e., $\bm{c}_{i}$, we preserve only the top $m$ values which are then modified to $1$ for a new binary similarity matrix $\bm{C}^{*}$.

Then, we extract each triplet from $\bm{C}^{*}$ by the following function:

[TABLE]

where $c^{*}_{xy}$ denotes the $xy$-th value of $\bm{C}^{*}$. Note each sample $\bm{x}$ can appear in many triplets. Therefore, we consider each $\bm{\tau}$ as a meta-element in the clustering, which improves the robustness due to the complementarity constraints.

Initializing Clusters: In the $I$-th iteration, we first determine an initial triplet (termed as $\bm{\tau}_{ini}^{I}$) from $\bm{T}_{out}$ to initialize the cluster $\mathcal{C}$, followed by merging the most correlated samples of $\bm{\tau}_{ini}^{I}$ into each $\mathcal{C}$.

Following [19], we initialize a new cluster using $\bm{\tau}_{ini}^{I}$ with highest local density:

[TABLE]

where $\rho$ calculates the local density defined in Definition 5. The high local density of the triplet reflects the most connections between $\bm{\tau}_{i}$ and other triplets, which produces the most connections between $\bm{x}_{\bm{n}_{j}^{i}}$ and other samples in $\bm{X}_{out}^{I}$.

Once the initialized triplet $\bm{\tau}_{ini}^{I}$ is determined, we iteratively extend the initialized cluster $\mathcal{C}$ by fusing the most confident triplets. For each triplet $\bm{\tau}_{i}$ in $\bm{T}_{out}$, we calculate the sum of the connection score regarding the samples in $\mathcal{C}$ to greedily determine whether the samples in $\bm{\tau}_{i}$ should be assigned into $\mathcal{C}$ or not:

[TABLE]

where $\bm{n},\bm{m}$ denote the set of indexes for the samples in $\bm{\tau}$ and $\mathcal{C}$, respectively. We iteratively update the auxiliary sets $\bm{T}_{out}^{I}$, $\bm{T}_{in}^{I}$, $\bm{X}_{out}^{I}$ and $\bm{X}_{in}^{I}$ in the iterations.

Terminating: We terminate the process of estimating the number of clusters and get $\widetilde{K}$ clusters if and only if $\bm{\tau}_{ini}^{\widetilde{K}+1}$ satisfies:

[TABLE]

Specifically, if the samples in $\bm{\tau}_{ini}^{\widetilde{K}+1}$ are of high frequency in $\bm{X}_{in}^{\widetilde{K}+1}$, i.e., the triplet with the highest local density in $\bm{T}_{out}^{\widetilde{K}+1}$ is already contained in $\bm{T}_{in}^{\widetilde{K}+1}$, we consider that the clusters are sufficient for modeling the intrinsic subspaces.

Avoiding Over-Segmentation: We also introduce an alternative step to check the redundancy among initialized clusters $\{\mathcal{C}_{i}\}_{i=1}^{\widetilde{K}}$ to avoid over-segmentation. We calculate the connection scores $s$ for small-scale clusters against others, and merge the highly correlated clusters $\mathcal{C}_{i}$ and $\mathcal{C}_{j}$ if we have

[TABLE]

where $|\mathcal{C}|$ denotes the number of samples in $\mathcal{C}$. We then get the initialized clusters $\{\mathcal{C}_{i}\}_{i=1}^{\widehat{K}}$, where $\widehat{K}$ is the estimated number of clusters and  $\widehat{K}\leq\widetilde{K}$.

Assigning Rest Samples: Given $\{\mathcal{C}_{i}\}_{i=1}^{\widehat{K}}$, we assign each of the remaining samples into $\mathcal{C}$ which evokes an optimal fusion reward. For $\bm{x}_{j}$, we find its optimal cluster $\mathcal{C}^{*}$ by the following equation:

[TABLE]

where $R_{f}(\mathcal{C}|\bm{x})$ is the fusion reward defined by (6).

III-F An Extension: Neighboring based AutoSC Algorithm

In Definition 1, we collect $m$ nearest neighbors according to the magnitude of similarities between sample $\bm{x}_{j}$ and all other samples in $\bm{X}_{-j}$. These similarities are depicted in $\bm{C}$ by optimizing (1) which is composed of a reconstruction loss term and a regularization term. In this subsection, we extend the autoSC with an alternative technique to find $\bm{N}_{m}(\bm{x}_{j})$ for each $\bm{x}_{j}$ based on greedy search.

For each data sample $\bm{x}_{j}$, we let $\mathcal{S}^{I}_{j}$ be the subspace spanned by $\bm{x}_{j}$ and its neighbors in the $I$-th iteration, where the neighbor set $\mathcal{N}_{j}$ is initialized as $\mathcal{N}^{0}_{j}=\varnothing$, and $\mathcal{S}^{I}_{j}\in\mathbb{R}^{D\times\text{dim}(\mathcal{N}_{j})}$. In each iteration, we measure the projected similarity between $\bm{x}_{j}$ and other non-neighbor samples by calculating the orthonormal ordinates in the spanned subspace. For example, to calculate the similarity between $\bm{x}_{j}$ and $\bm{x}_{k}$ in the $I$-th iteration, we have

[TABLE]

where $\lVert\cdot\rVert_{F}^{2}$ denotes the Frobenius norm and $\bm{x}_{k}\in\bm{X}_{-\mathcal{N}}$. Consequently, for $\bm{x}_{j}$ in the $I$-th iteration, we find the closest neighbor and update $\mathcal{S}_{j}$ as follows:

[TABLE]

Here, we find one neighbor in each iteration and update the spanned subspace accordingly. The newly spanned subspace reflects more local structure of the ambient space which is assumed to cover the current sample. The neighbor set $\mathcal{N}_{j}$ is also updated by adding the new neighbor which is found in the $I$-th iteration. Finally, with $m$ iterations for each sample, we get an alternative $m$ nearest neighbor set $\mathcal{N}$.

Given the neighbor matrix $\mathcal{N}$, we propose the neighboring based autoSC algorithm (autoSC-N) to directly discover the triplet relationship among data samples, followed by optimizing both model selection and fusion rewards for clustering. The main steps of autoSC-N are summarized in Algorithm 2.

III-G Computational Complexity Analysis

In traditional subspace clustering system, the calculation of self-representation requires solving $N$ convex optimization problems over $D\times N$ constraints [20]. Spectral clustering is based on an eigen-decomposition operation on the Laplacian matrix followed by conducting K-means on the eigenvectors, both of which are time-consuming, involving a complex algebraic decomposition and iterative optimization, respectively [68, 71]. The overall computational complexity can be more than $O(N^{3})$. For the proposed autoSC, it takes $O(Nm^{2})$ to collect the triplet relationships for $N$ samples in the space spanned by the $m$ nearest neighbors. Here, since we have $m\ll N$, the complexity of collecting the triplet relationships is $O(N)$. The optimization of both model selection and fusion rewards takes $O(Nn)$ where the number of triplets $n$ has the same order of magnitude as $N$. Specifically, we have $n\approx N\times\frac{m}{2}$, and thus the complexity of clustering is $O(N^{2})$.

For the extension, i.e., autoSC-N, collecting the neighbor matrix takes $O(Nm)$ where the basic operation is a dot product of the $D$-dimensional tensors. This avoids the calculation of any convex optimization problem.

IV Experiments

IV-A Experimental Setup

In the experiments, we compare the automatic methods on the benchmark datasets, i.e., the extended Yale B [72] and the COIL-20 [73] dataset, followed by verifying the robustness of the proposed method to different $\bm{C}$ derived from various self-representation schemes along with combinations of different methods for estimating the number of clusters and segmenting the samples. We design comprehensive evaluation metrics to validate the clustering performance, i.e., the error rate of the number of clusters and the triplets. For all experiments on subsets, the reported results are the average of $50$ trials. We also conduct experiments on a motion segmentation task using the Hopkins 155 dataset.

IV-A1 Datasets

The extended Yale B [72] dataset is a widely used face clustering dataset which contains face images with different illumination of $38$ subjects, each subject has $64$ images.

The COIL-20 [73] dataset consists of $20$ different real subjects, including cups, bottles and so on. For each subject, there are $72$ images with different camera viewpoints.

The Hopkins 155 dataset [74] consists of $155$ video sequences. For each video sequence, there are $2$ or $3$ motions.

IV-A2 Comparative Methods

We make comparisons with the following methods: SCAMS [13, 29], a density peak based method (DP) [19], a singular value decomposition based method (SVD) [58] and DP-space [12]. Besides, we utilize the following subspace representation methods to generate different coefficient matrices $\bm{C}$: LRR [39], CASS [52], LSR [46], SMR [32] and ORGEN [33]. The similarity matrix $\bm{C}$ is then used to calculate the triplet relationships for autoSC.

IV-A3 Evaluation Metrics

To evaluate the performance of the proposed triplets, we define the error rate $\mathcal{A}$ as follows:

[TABLE]

where $n$ denotes the number of the triplets and $\sigma(\bm{\tau}|\bm{g}^{*})$ is the counting function on the frequency that $\bm{x}\in\bm{g}^{*}$ for all $\bm{x}\in\bm{\tau}$. Here the output of $f$ ranges from [math] to $2$. The dynamic set $\bm{g}_{i}^{*}$ consists of samples in one subspace $S$ according to the ground truth, where $S$ contains as many samples in $\bm{\tau}_{i}$ as possible.

We introduce the error rate of the number of clusters (NCe) as the primary evaluation metric for the clustering methods which estimate the number of clusters $\widehat{K}$ automatically:

[TABLE]

where $K$ is the real number of clusters, $M$ is the number of trials and $\widehat{K}_{i}$ is the estimated number of clusters in the $i$-th trial. We also use the standard normalized mutual information (NMI) [75] to measure the similarity between two clustering distributions, i.e., the prediction and the ground truth. With respect to NMI, the entropy illustrates the nondeterminacy of one clustering to the other, and the mutual information quantifies the amount of information that one variable obtains from the other.

IV-A4 Parameter $m$

The parameter $m$ in Definition 1, i.e., the number of preserved neighbors for each sample, is related to the intrinsic dimension of the subspaces. We empirically evaluate the influence of $m$ on both extended Yale B and COIL-20 datasets with 15 subjects. Besides, we use subspace representations derived from SMR. The results are shown in Table III. As shown in the table, the proposed method achieves best performance when we have $m=8$ for most cases. Actually, the parameter $m$ is robust since the performance is stable when $m>8$.

IV-B Comparisons among Automatic Clustering

We conduct experiments on the extended Yale B and COIL-20 datasets with different numbers of subjects, and compare four methods with the proposed autoSC and autoSC-N on the metrics of NCe and NMI. For SCAMS [13, 29], DP [19], SVD [58] and our autoSC, the optimization module in SMR [32] is employed to generate the similarity matrix $\bm{C}$. The DP-space method simultaneously estimates $\widehat{K}$ and finds the subspaces without the requirement of a similarity matrix. All parameters of the contrasted methods are tuned to provide the best performance.

Fig. 2 and Table II report the performance. As shown in Table II, when combining SMR, the averaged NCe of autoSC is smaller than other comparative methods on all experimental configurations, indicating that it gives a close estimation on the number of clusters. For example, the estimated $\widehat{K}$ on the extended Yale B with $8$ subjects has a deviation of less than $1$, and produces a NMI higher than $0.9$. The autoSC-N gets the second best performance on most configurations, which demonstrates the effectiveness of both triplet relationship and reward optimization. In contrast, SVD achieves comparable results on the small-scale configuration of each dataset, but the performance becomes poor when the number of samples increases. It is mainly because the largest gap between the pair of singular values decreases when the number of clusters becomes larger. When combining SMR, SCAMS performs comparably according to NMI on both datasets, however, as is illustrated in Fig. 2 (a) and Table II, it provides a much larger $\widehat{K}$ than the ground truth, e.g., $\widehat{K}>100$ when $K=30$ on the extended Yale B dataset. NMI does not strongly penalize over-segmentation, making the metric NCe be the primary evaluation of the SCAMS method. The DP-space performs well on NCe, but has poor performance on the NMI. This is because most samples are assigned into one cluster, and the other clusters are small. In addition, when combining LSR, as shown in Table II, the performance of all methods decrease, while the proposed autoSC still achieves best performance on most configurations. It demonstrates the generalization ability of our autoSC.

IV-C Robustness to Self-Representations Schemes

The methods including SCAMS [13, 29], DP [19], SVD [58] and the proposed autoSC require the similarity matrix $\bm{C}$ as input. Also, for DP [19], the distance among samples needs to be calculated. We calculate the distance $d_{ij}$ between samples $\bm{x}_{i}$ and $\bm{x}_{j}$ by $d_{ij}=\frac{1}{c_{ij}}$ rather than the simple Euclidean distance. To verify the robustness of the proposed autoSC regarding various subspace representations, we calculate the similarity matrix $\bm{C}$ using $5$ subspace representation modules, followed by the combinations with the $4$ methods which automatically estimate the number of clusters and segment the samples.

Table IV shows the evaluation results of $\mathcal{A}$ on both datasets with the combinations of $5$ subspace representations, while the NCe and NMI on the extended Yale B dataset with $8$ subjects are reported in Fig. 3. Moreover, we visualize the similarity matrix $\bm{C}$ derived from $5$ subspace representation modules in Fig. 4. We can see from Fig. 3 that the SCAMS, DP and SVD methods are sensitive to the choice of the subspace representation module. For example, DP estimates $\widehat{K}$ as a relatively close value to the ground truth when combined with SMR (NC${}_{e}=3.06$), but generates a totally wrong estimation when combined with LRR (NC${}_{e}=265.60$). Different subspace representation modules generate coefficient matrices with various intrinsic properties [4], thus the parameter for truncation error $\epsilon$ needs to be tuned carefully.

For the proposed autoSC, it is stable on different combinations considering the metric of NCe and $\mathcal{A}$, which demonstrates the complementary ability of the proposed method. For all combinations, the error rate of the triplets obtained from (10) is less than $2\%$, which guarantees the consistency of the proposed autoSC with different kinds of $\bm{C}$. Furthermore, it shows better performance when combined with CASS, LSR and SMR than other combinations on both metrics in Fig. 3. The reason lies on the guarantee of the mapping invariance which is termed as the grouping effect [52, 46, 32], together with the filtering of weak connections and the self-constraint among samples within triplets. As shown in Fig. 4 (b), (c), (d), the coefficient matrices are dense while it shows block-diagonal structure in Fig. 4 (d) and each block corresponds to one cluster. Therefore, the nearest neighbors which are used to generate the triplets can be chosen precisely. The performance decreases when combined with ORGEN since the similarity matrix $\bm{C}$ derived from ORGEN is sparse with less locations for constructing effective triplets.

IV-D Time Efficiency

Table V shows the run-time of comparative methods using subsets from the extended Yale B dataset. The experiments are conducted on a machine with a $2.93$GHz CPU and $32$GB RAM. AutoSC-N requires the least run-time compared to all comparative methods due to the following two reasons. First, autoSC-N explores the neighborhood relationship in the raw data space rather than solving a convex optimization problem. Second, it employs a greedy optimization scheme to estimate the number of clusters and calculate the clustering assignment rather than a complex optimizing method such as computing the singular value decomposition. Note the proposed autoSC method achieves the second best result among comparative methods.

IV-E Real Application: Motion segmentation

Motion segmentation refers to the task of segmenting multiple video sequence. The candidate video is composed of multiple foreground objects, which are rapidly moving and required to be clustered into spatiotemporal regions corresponding to specific motions. Following the traditional scheme [16], we consider the Hopkins 155 dataset [74] and solve the motion segmentation problem by first extracting a set of feature points for each frame followed by clustering them based on the motions. Table VI reports the comparison against four automatic clustering methods. For SCAMS [13, 29], DP [19] and SVD [58], the SMR [32] is firstly conducted to calculate the similarity matrix. As shown in the table, the proposed autoSC achieves best performance on both metrics, indicating that the autoSC is effective at both estimating the number of motions (about $0.18$ error rate) and segmenting the feature points (obtains NMI of more than $0.98$). In addition, it shows favorable efficiency on the motion segmentation task. The autoSC-N is the most efficient method ($0.26s$ per sequence) with second best performance on NCe and NMI. The SVD method obtains the best result among other comparative methods, but it consumes much more time (about more than $2.5s$ per sequence) due to the singular value decomposition process.

V Conclusion

In this paper, we propose a joint model to estimate the number of clusters and segment the samples in a data set. Based on the self-representation of dataset, we first design a hyper-correlation oriented meta-element termed as the triplet relationship, which indicates a compact local structure among three samples. The triplet is more robust than pairwise relationships when partitioning samples near the intersection of two subspaces due to the complementarity of mutual restrictions. Accordingly, we propose the autoSC method to optimize two reward functions simultaneously, of which the model selection reward constrains the number of clusters and the fusion reward facilitates the clustering assignment of the samples. Both functions are greedily maximized during the clustering process. In addition, we provide an extension of autoSC which automatically calculates the neighboring relationship in the raw data space rather than a similarity space spanned by self-representation. Experimental results on face clustering, synthetic dataset clustering and motion segmentation tasks demonstrate the effectiveness and efficiency of our approaches.