A parameter-free graph reduction for spectral clustering and SpectralNet

TL;DR

This paper presents a parameter-free graph reduction technique that enhances spectral clustering and SpectralNet by automatically constructing stable graphs without the need for parameter tuning.

Contribution

A novel parameter-free graph reduction method that improves spectral clustering and SpectralNet stability across datasets.

Findings

The method produces stable clustering results without parameter tuning.

It outperforms parameter-dependent methods in stability across various datasets.

Experiments demonstrate consistent performance where other methods fluctuate.

Abstract

Graph-based clustering methods like spectral clustering and SpectralNet are very efficient in detecting clusters of non-convex shapes. Unlike the popular $k$-means, graph-based clustering methods do not assume that each cluster has a single mean. However, these methods need a graph where vertices in the same cluster are connected by edges of large weights. To achieve this goal, many studies have proposed graph reduction methods with parameters. Unfortunately, these parameters have to be tuned for every dataset. We introduce a graph reduction method that does not require any parameters. First, the distances from every point $p$ to its neighbors are filtered using an adaptive threshold to only keep neighbors with similar surrounding density. Second, the similarities with close neighbors are computed and only high similarities are kept. The edges that survive these two filtering steps form the constructed graph that was passed to spectral clustering and SpectralNet. The experiments showed that our method provides a stable alternative, where other methods performance fluctuated according to the setting of their parameters.

Equations

$$A_{pq}=\exp \left(\frac{-d^2\left(p,q\right)}{{\sigma }_p{\sigma }_q}\right).$$

$$MW{A}_i=\frac{v_{i-1}+v_i+v_{i+1}}{r_{i-1}+r_i+r_{i+1}},$$

$$(p,q)\in E(G)\iff A_{pq}>T_p.$$

$$(p,q)\in E(G)\iff A_{pq}>T_p\text{and}{A}_{qp}>T_q.$$

$$ACC(T,L)=\frac{{\sum }_{i=1}^N\delta (T_i,map(L_i))}{N},$$

$$ARI(T,L)=\frac{2(n_{00}{n}_{11}-n_{01}{n}_{10})}{(n_{00}+n_{01})(n_{01}+n_{11})+(n_{00}+n_{10})(n_{10}+n_{11})}.$$

$$E\%=\frac{E(G_{reduced})}{E(G_{full})}.$$

Full text

1 Introduction

The problem of detecting clusters of non-convex geometric shape, has been long studied in the literature of pattern recognition. The solutions of this problem could be broadly classified into two categories: kernel- and graph-based methods. Kernel-based methods attempt to map the points into a space where they can be separated. The embedding function $\phi:\mathbb{R}^{D}\rightarrow\mathbb{R}^{M}$ maps points from the original space to an embedding space. Defining the embedding function $\phi$ is usually unknown and could be computationally expensive [22]. On the other hand, graph-based methods use the graph $G(V,E)$ whose set of vertices represents the data points and its set of edges represents the similarity between each pair of vertices. Finding non-convex clusters in a graph could be done in three ways: 1) by iteratively coarsening and partitioning the graph [12], 2) by performing spectral clustering [1], and 3) by feeding the graph $G(V,E)$ to a neural network (SpectralNet) [24]. The first way of detecting clusters in a graph is iterative which involves two deficiencies: the risk of being trapped in a local minima and the need for a stopping condition. This makes spectral clustering and SpectralNet more appealing for studies conducting graph-based clustering.

Spectral clustering starts by constructing a graph $G(V,E)$. The sets of vertices $V$ and edges $E$ represent data points and their pairwise similarities. Spectral clustering detects clusters by performing eigen-decomposition on the graph Laplacian matrix $L$ and running $k$-means on its top eigenvectors [18]. The computational bottleneck represented by the eigen-decomposition would cost the algorithm computations in the order of $\mathcal{O}(N^{3})$ [31]. This stimulated the research on reducing these computations by reducing the graph vertices and/or edges. However, the need for a memory efficient graph creates another problem related to the number of parameters associated with the process of graph construction. Deciding the number of reduced vertices and how the edges are eliminated would create several parameters that need careful tuning.

SpectralNet [24] uses Siamese nets to learn affinities between data points. Then it feeds these affinities to a neural network to find a map function $F_{\theta}$, which maps the graph vertices $V$ to an embedding space where they can be separated using $k$-means. The Siamese nets expect the user to label which pairs are positive (similar) and which are negative (dissimilar). An unsupervised pairing uses the $k$-nearest neighbors, where the nearest neighbors are the positive pairs and the farthest neighbors are the negative pairs. The parameter $k$ requires manual tuning. It also restricts the number of edges to be exactly $k$, regardless of the surrounding density around the data point.

In her spectral clustering paper, von Luxburg [16] wrote about the advantages of mutual k-nearest neighbor graph, and how it “tends not to connect areas of different density”. She highlighted the need for having a “heuristic to choose the parameter k”. We introduce a graph reduction method that does not require any parameters to produce a mutual graph with a reduced number of edges compared to the size of the full graph $E=N\times N$, where $N$ is number of all vertices. It initially finds the mean distance that best describes the density around a point. Then, it computes the pairwise similarities based on: 1) the distance between a pair of points, and 2) the mean distance of the surrounding density. Finally, we construct a mutual graph where a pair of vertices must be in each other’s nearest neighbors sets. We used two graph applications for the experiments: spectral clustering and SpectralNet. The proposed method provides a stable alternative compared to other methods where their performance was determined by the selected parameters.

Our main contribution in this work, is eliminating manually tuning parameters that affect the clustering accuracy when changed. The graph partitioning methods used in this work are spectral clustering [16] and SpectralNet [24].

2 Related work

The problem of detecting non-convex clusters has led to the development of numerous clustering methods. These methods have abandoned the assumption that a cluster has a single mean. Instead, they rely on pairwise similarities to detect clusters. Graph-based clustering involves two steps: 1) reducing the graph, and 2) partitioning the graph. The proposed method in this paper falls under graph construction methods.

Spectral clustering uses eigen-decomposition to map the points into an embedding space, then groups similar points. One of the important application of spectral clustering is subspace clustering [15, 8]. The performance of spectral clustering is determined by the similarity metric used to construct the affinity matrix $A$. The earlier works of subspace clustering used affinities based on principal angles [30]. But recent studies have used sparse representation of points to measure the similarity [15, 20, 21]. Spectral clustering requires computations in order of $\mathcal{O}(N^{3})$, due to the eigen-decomposition step. A straightforward solution to this problem is to reduce the size of the affinity matrix $A$. This can be done in two ways: 1) reducing the set of vertices $V$, or 2) reducing the set of edges $E$.

Reducing the number of vertices is done by placing representatives on top of the data points, and then using those representatives as graph vertices. Placing a representative could be done by sampling (like $k$-means++ [3]) or by vector quantization (like self-organizing maps [13]). A well-known method in this field is “k-means-based approximate spectral clustering (KASP)” proposed by Yan et al. [31]. KASP uses $k$-means to place representatives. Other efforts by Tasdemir [28] and Tasdemir et al. [29] involved placing representatives using vector quantization, and a nice feature of these methods is that the pairwise similarities are computed during the vector quantization. The problem with these methods is the parameter $m$, which is the number of representatives. Specifically, how should we set $m$? And how would different values of $m$ affect the clustering results?

Reducing the graph edges could be done by setting the neighborhood conditions. For instance, let $p$ be the center of a ball $B(p,r)$ with radius $r$ and $q$ be the center of a ball $B(q,r)$. $p$ and $q$ are connected if and only if the intersection of $B(p,r)$ and $B(q,r)$ does not contain other points [17]. Such graphs are called Relative Neighborhood Graphs (RNGs). Correa and Lindstrom [7] used a $\beta$-skeleton graph for spectral clustering. However, the parameter $\beta$ needs tuning. Alshammari et al. [1] introduced a method to filter edges from a $k$-nearest neighbor graph. However, it still needs an influential parameter, which was the mean of the baseline distribution of distances $\mu_{0}$. Another local method to reduce the number of edges was proposed by Satuluri et al. [23]. The authors measured the similarity between two vertices using adjacency lists overlap, a metric known in the literature as shared nearest neighbor similarity [11]. A graph sparsification method based on effective resistance was proposed by Spielman and Srivastava [25, 26]. Their method was theoretically solid, but the definition of effective resistance breaks the cluster structure of the graph. Vertices with more short paths have low effective resistance and the method disconnects them. Retaining the cluster structure of the graph requires connecting such vertices [23].

In spectral clustering, the obtained spectral embedding cannot be extended to unseen data, a task commonly known as out-of-sample-extension (OOSE). Several studies have proposed solutions to this problem. Bengio et al. [4] used Nyström method to approximate the eigenfunction for the new samples. But they have to check the similarity between the training and new samples [19]. Alzate and Suykens [2] proposed binarizing the rows of eigenvectors matrix where each row corresponds to a single training data point. By counting row occurrences, one can find the $k$ most occurring rows, where each row represents an encode vector for a cluster. To label a test sample, its projection was binarized and it is assigned to the closest cluster based on the minimum Hamming distance between its projection and encoding vectors. Levin et al. [14] proposed a linear least squares OOSE, which was very close to Bengio et al. [4] approach. They also proposed a maximum-likelihood OOSE that produces a binary vector $\overrightarrow{a}$ indicating whether the unseen sample has an edge to the training samples or not.

All previous methods that provided an out-of-sample-extension (OOSE) to spectral clustering have relied on eigen-decomposition, which becomes infeasible for large datasets. The newly proposed SpectralNet [24] is different from spectral clustering in a way that it does not use eigen-decomposition step. Instead, SpectralNet passes the affinity matrix $A$ into a deep neural network to group points with high similarities. Yet, SpectralNet still needs a graph construction method. Previous SpectralNet works have used $k$-nearest neighbour graph, but they have to set the parameter $k$ manually. It also restricts the number of edges to be exactly $k$, regardless of the surrounding density around the data point. Dense clusters require more edges to be strongly connected. Strong connections ensure closer positions in the embedding space. Also, SpectralNet methods randomly choose the negative pairs. This random selection makes the method inconsistent in independent executions.

Considering the literature on reduced graphs for spectral clustering and SpectralNet, it is evident that they have two deficiencies. First, certain parameters are required to drive the graph reduction process. Second, the involvement of random steps makes these methods inconsistent over independent executions.

3 Reducing the graph size without the need for parameters

The motivation behind our work was to avoid the use of any parameters during the graph reduction. The input for our method is a $k$-nearest neighbor graph. Although this $k$-nn graph is sparsified, it still connects clusters with different densities. The value of $k$ has limited influence on the final graph because it is not final, and most of the unnecessary edges created by $k$-nn will be removed in the reduction process. The method starts by finding the value of $\sigma_{p}$ that best describes the local statistics around a randomly chosen point $p$. Then, it filters the edges with low weights. Finally, it checks the mutual agreement for each edge.

3.1 Finding the value of $\sigma_{p}$

To compute pairwise similarities, we used the similarity measure introduced by [32], which is defined as follows:

[TABLE]

where $-d^{2}\left(p,q\right)$ is the distance between points $p$ and $q$. $\sigma_{p}$ and $\sigma_{q}$ are the local scales at points $p$ and $q$ respectively. What is good about this similarity measure is that it uses two sources of information to compute the pairwise similarity: 1) the distance between them, and 2) the surrounding density for each point. Points belonging to clusters with different densities would have a low similarity even if they are separated by a small distance This makes this measure superior for highlighting different clusters separated by a small distance.

One problem that arises from using this measure in equation 1 is how to set the value of $\sigma_{p}$ in the denominator. In previous studies, it was set as the distance to the 7th neighbor [32, 27]. However, there is no evidence that the distance to the 7th neighbor would work in every dataset. Using the data to select this parameter would be more practical.

The idea behind the parameter $\sigma_{p}$ is to measure the sparseness of a cluster. If $p$ lies in a sparse cluster, it would have a large $\sigma_{p}$; whereas if $p$ lies in a dense cluster, it would have a small $\sigma_{p}$. To achieve this, we need to exclude neighbors with different local density than $p$ from being included in computing $\sigma_{p}$. We used a smooth histogram of distances to characterize the local density for $p$ neighbors (as shown in Figure 1). The intuition is if a neighbor has different local density than $p$, this would be represented as a peak on the histogram. The histogram bin values for each point are smoothed using the moving weighted average (MWA). The smoothing was designed as follows:

[TABLE]

where $v$ is the value of the bin and $r$ is the rank of the bin, with $r=1$ being the bin containing the closest points to the point $p$. This smoothing assigns weights to the bins based on their distance from $p$, with high weights assigned to closer bins and low weights assigned to bins further away.

The histogram threshold tells us that up to the Kth neighbor, the local density of $p$ has not changed. Then, we compute $\sigma_{p}$ as the mean distance from the 1st to the Kth neighbor. This process is described in statements 4 to 9 in Algorithm 1.

3.2 Reducing the graph edges

Once we have $\sigma_{p}$ for each point, we can calculate the pairwise similarities using the formula in equation 1, as shown in statements 10 to 14 in Algorithm 1. Large values indicate highly similar points, whereas low values indicate dissimilarity. We build another histogram of all the pairwise similarities using the Freedman–Diaconis rule [9] as shown in Figure 2. For each point, similarities lower than the threshold $T_{p}$ are eliminated. If the maximum similarity is larger than the mean plus the standard deviation $\mu+\sigma$, the threshold is set as $T=\mu+\sigma$. If not, the threshold is set as $T=\mu-\sigma$. Figure 2 shows the included similarities as blue bins and the excluded similarities as red bins. The graph edges are defined as:

[TABLE]

where $(p,q)$ is the edge between points $p$ and $q$. $A_{pq}$ is the weight assigned to the edge $(p,q)$. This process is described in statements 15 to 21 in Algorithm 1.

The last step of our reduction method is to build a mutual graph. In a mutual graph, a pair of points should agree to accept an edge. This makes the graph $G$ to be defined as:

[TABLE]

where $T_{p}$ is threshold of acceptance for the vertex $p$.

3.3 Integration with SpectralNet

Our graph filtering method can be seamlessly integrated to the newly proposed spectral clustering using deep neural networks (SpectralNet) [24]. SpectralNet uses Siamese nets [5] to learn affinities between data points. Siamese nets expect the user to label which pairs are positive and which are negative. An unsupervised pairing uses the $k$-nearest neighbors, where the nearest neighbors are positive pairs and the farthest neighbors are negative pairs. Our graph filtering can be used to obtain positive and negative pairs. It offers the advantage of setting the number of pairs per point dynamically. This cannot be achieved using $k$-nearest neighbors, where all the points are restricted to have a fixed number of positive pairs. Also, we do not have to set $k$ manually. We let our method assigns the positive and negative pairs for each point. A pseudocode illustrating the steps of the proposed method is shown in Algorithm 1.

4 Experiments and discussions

In the experiments we used four synthetic datasets, as shown in Figure 3. Dataset 1 to 3 were created by [32], while Dataset 4 was created by us. We also used seven real datasets (see Table 1). Apart from the MNIST dataset, all the real datasets were retrieved from UCI machine learning. Each dataset was run with two parameter sets to evaluate the effect.

Six methods were used for comparison and these are shown in Table 2. Methods 1 to 5 [31, 28, 29] rely on the parameter $m$, which is the number of representatives to build the graph $G$. They used iterative algorithms like $k$-means and self-organizing maps to construct the graph, which makes them produce a slightly different graph with each run. Method 6 [1] relied on the parameter $\mu_{0}$ to build the graph $G$, where $\mu_{0}$ is the number of neighbors whose mean was used as a threshold to include or exclude further neighbors. The code is available at https://github.com/mashaan14/Spectral-Clustering.

All the methods were evaluated using three evaluation metrics: 1) clustering accuracy (ACC) 2) the Adjusted Rand Index (ARI) [10], and 3) the percentage of edges used compared to all the edges in a full graph (E%).

ACC computes the percentage of hits between ground truth labels $T_{i}$ and labels obtained through clustering $L_{i}$. It is defined as [6]:

[TABLE]

where $N$ is the number of points and the function $\delta(x,y)$ is the Kronecker delta function, which equals one if $x=y$ and zero otherwise. The function $map(L_{i})$ permutes the grouping obtained through clustering for the best fit with the ground-truth grouping. ARI needs two groupings $T$ and $L$, where $T$ is the ground truth and $L$ is the grouping predicted by the clustering method. If $T$ and $L$ are identical, ARI produces one, and zero in case of random grouping. ARI is calculated using: $n_{11}$: pairs in the same cluster in both $T$ and $L$; $n_{00}$: pairs in different clusters in both $T$ and $L$; $n_{01}$: pairs in the same cluster in $T$ but in different clusters in $L$; $n_{10}$: pairs in different clusters in $T$ but in the same cluster in $L$.

[TABLE]

The computational efficiency can be measured by the method’s running time, but, this is influenced by the type of machine used. We chose to measure the computational efficiency by the percentage of edges, E%:

[TABLE]

4.1 Experiments on synthetic data

In the synthetic datasets the proposed method delivered a performance that ranked it as 2nd, 2nd, 1st, and 2nd for Datasets 1 to 4 respectively (see Figure 4). Method 6 was the top performer on three occasions. However, its performance dropped significantly when we changed the parameter $\mu_{0}$. For example, its performance dropped by 50% with Dataset 4 when we changed $\mu_{0}=3$ to $\mu_{0}=7$. This shows how parameters could affect the performance. Another observation is the consistency of ACC and ARI metrics over the 50 runs. By looking at Figure 4, the methods 1 to 5 have a wide standard deviation. This is explained by the iterative algorithms used by methods 1 to 5 to construct the graph. Method 6 and the proposed method do not have this problem, and they have a small standard deviation. This is due to their deterministic nature when constructing the graph, which makes them consistent over independent executions.

In terms of the used edges, the proposed method used 6.32%, 1.45%, and 0.51% of the full graph edges for Datasets 2 to 4 respectively. But in Dataset 1 there was a sharp increase where the proposed method used 16% of the full graph edges. This sharp increase could be explained by the points in dense clusters being fully connected.

4.2 Experiments on real data

With real datasets in Figure 5, the proposed method continued to be the most consistent method over all tested methods. It kept a very small standard deviation, while other methods had a wide standard deviation. The performance of other methods was determined by their parameters. For example, Method 3 was the best performer on iris dataset when $m=16$. However, when we changed $m$ to $32$ , its performance dropped by more than 15%. Another observation with statlog and MNIST the proposed method did not perform well. This indicated that a cluster in these datasets does not have the same statistics across its regions. Therefore, characterizing clusters using local $\sigma$ might not be a good choice. Instead, we should use CONN to discover clusters discontinuity, rather than tracking local statistics.

4.3 Effect of the parameters on the spectral clustering performance

In this experiment, we investigated how a wide selection of parameters could affect the accuracy of the spectral clustering. The parameters $m$ and $\mu_{0}$ were given the following values: $m\in\{10,20,30,40,50,60,70,80,90,100\}$ and $\mu_{0}\in\{3,7,10,20,30,40,50,60,70,80\}$. In Figure 6 (left), the performance Methods 1 to 5 fluctuated with different values of $m$, with a clear downward trend seen as $m$ increased. The dashed horizontal line is the performance of the proposed method. In Figure 6 (right), Method 6 started with low performance, peaking around $\mu_{0}=30$, and then it took a downward trend. By eliminating the use of $\mu_{0}$, our method delivered a stable performance as shown by the horizontal dashed line.

4.4 Experiments for integration with SpectralNet

The SpectralNet integration experiment was conducted using three datasets shown in Figure 7. The evaluation metrics are ACC shown in equation 5, ARI shown in equation 6, and total pairs, which is the number of pairs passed to the Siamese net. We used four methods to construct positive and negative pairs. The first two methods used a $k$-nearest neighbor graph with $k=2$ and $k=4$. Simply, the nearest $k$ neighbors were set as the positive pairs, and $k$ random farthest neighbors were set as the negative pairs. The third method used the parameter $\mu_{0}$ proposed by Alshammari, et al. [1] to construct pairs.

In Figure 8, the proposed method delivered the best performance for the cc and compound datasets. This good performance was coupled with good computational efficiency, with an average of $8,468$ for the total pairs passed to the Siamese net. Only $k=2$ could deliver fewer total pairs, but with a massive loss in performance. For the aggregation dataset, $k=2$ delivered the best performance. This experiment highlighted the need for setting the number of positive pairs dynamically. The methods following this approach (the $\mu_{0}$ method and our method) were the best performers for two of the three datasets.

5 Conclusion

The problem of detecting non-convex clusters has led to the development of numerous clustering methods. One of the well-known graph-based clustering methods is spectral clustering and SpctralNet. Both spectral clustering and SpectralNet require a graph that connects points in the same cluster with edges of high weights. The intuition is simple, strongly connected points will become closer in the embedding space and can be easily detected.

Graph reduction requires extensive use of parameters that need careful setting for each dataset. The graph reduction algorithm proposed in this study does not require any parameters to reduce the graph, yet it is able to maintain spectral clustering and SpectralNet accuracies. It takes an input as a full graph or a $k$-nearest neighbor graph (in the case of a large number of points). Then, it reduces the graph edges using statistical measures that require low computations. The experiments revealed that the proposed method provides a stable alternative compared to other methods that require parameters tuning.

The proposed method does not reduce the graph vertices, which could boost the computational efficiency. A useful extension of the proposed method would be a vertices reduction component that is aware of local statistics. Another potential improvement of this work is to use a different kernel other than gaussian kernel to compute pairwise similarities.