Graph sketching-based Space-efficient Data Clustering

TL;DR

This paper introduces DBMSTClu, a space-efficient, non-parametric clustering method that uses a sketched MST from limited measurements to accurately identify arbitrary-shaped clusters on resource-constrained devices.

Contribution

It presents a novel MST-based clustering algorithm that operates under strict space constraints using streaming sketches, without requiring input parameters.

Findings

Effective at identifying non-convex clusters.

Requires significantly less memory than traditional methods.

Outperforms state-of-the-art on multiple datasets.

Abstract

In this paper, we address the problem of recovering arbitrary-shaped data clusters from datasets while facing \emph{high space constraints}, as this is for instance the case in many real-world applications when analysis algorithms are directly deployed on resources-limited mobile devices collecting the data. We present DBMSTClu a new space-efficient density-based \emph{non-parametric} method working on a Minimum Spanning Tree (MST) recovered from a limited number of linear measurements i.e. a \emph{sketched} version of the dissimilarity graph $\mathcal{G}$ between the $N$ objects to cluster. Unlike $k$-means, $k$-medians or $k$-medoids algorithms, it does not fail at distinguishing clusters with particular forms thanks to the property of the MST for expressing the underlying structure of a graph. No input parameter is needed contrarily to DBSCAN or the Spectral Clustering method. An approximate MST is retrieved by following the dynamic \emph{semi-streaming} model in handling the dissimilarity graph $\mathcal{G}$ as a stream of edge weight updates which is sketched in one pass over the data into a compact structure requiring $O(N \operatorname{polylog}(N))$ space, far better than the theoretical memory cost $O(N^2)$ of $\mathcal{G}$. The recovered approximate MST $\mathcal{T}$ as input, DBMSTClu then successfully detects the right number of nonconvex clusters by performing relevant cuts on $\mathcal{T}$ in a time linear in $N$. We provide theoretical guarantees on the quality of the clustering partition and also demonstrate its advantage over the existing state-of-the-art on several datasets.

Figures

Tables

Table 1. Table 1: Silhouette coefficients, ARI and DBCVI for the noisy circles (left) and noisy moons (right) datasets.

	Silhouette coeff.		ARI		DBCVI
SEMST	0.16	-0.12	0	0	0.001	0.06
DBSCAN	0.02	0.26	0.99	0.99	-0.26	0.15
DBMSTClu	-0.26	0.26	0.99	0.99	0.18	0.15

Table 2. Table 2: Numerical values for DBMSTClu’s execution time (in s) varying N 𝑁 N and K 𝐾 K (avg. on 5 5 5 runs). The last row shows the execution time ratio between K = 100 𝐾 100 K=100 and K = 5 𝐾 5 K=5 .

$K\backslash N$	$1000$	$10000$	$50000$	$100000$	$250000$	$500000$	$750000$	$1000000$
$5$	$0.34$	$2.96$	$14.37$	$28.91$	$73.04$	$148.85$	$218.11$	$292.25$
$20$	$0.95$	$8.73$	$43.71$	$88.51$	$223.18$	$449.37$	$669.29$	$889.88$
$100$	$4.36$	$40.25$	$201.76$	$398.41$	$995.42$	$2011.79$	$3015.61$	$4016.13$
“$100/5$”	$12.82$	$13.60$	$14.04$	$13.78$	$13.63$	$13.52$	$13.83$	$13.74$

Table 3. Table 3: Silhouette coefficients, Adjusted Rand Index and DBCVI for the blobs, noisy circles and noisy moons datasets with SEMST, DBSCAN and DBMSTClu.

	Silhouette coeff.			Adjusted Rand Index			DBCVI
SEMST	0.84	0.16	-0.12	1	0	0	0.84	0.001	0.06
DBSCAN	0.84	0.02	0.26	1	0.99	0.99	0.84	-0.26	0.15
DBMSTClu	0.84	-0.26	0.26	1	0.99	0.99	0.84	0.18	0.15

Equations

$$\forall i\in[K],\ \operatorname{DISP}(C_{i})=\left\{\begin{array}[]{ll}\max\limits_{j,\ e_{j}\in C_{i}}w_{j}&\mbox{if }\ |E(C_{i})|\neq 0\\ 0&\mbox{otherwise.}\end{array}\right.$$

$$\forall i\in[K],\ \operatorname{SEP}(C_{i})=\left\{\begin{array}[]{ll}\min\limits_{j,\ e_{j}\in Cuts(C_{i})}w_{j}&\mbox{if }\ K\neq 1\\ 1&\mbox{otherwise.}\end{array}\right.$$

$$V_C(C_i)=\frac{\mathrm{SEP}(C_i)-\mathrm{DISP}(C_i)}{\max (\mathrm{SEP}(C_i),\mathrm{DISP}(C_i))}$$

$$\mathrm{DBCVI}(\Pi )=\sum _{i=1}^K\frac{|C_i|}{N}{V}_C(C_i)$$

$$w_2>\frac{2n_2{w}_1-n_1+\sqrt{n_1^2+4w_1(n_2^2{w}_1+N^2-N{n}_1-n_2^2)}}{2(N-n_1+n_2)}$$

$$\frac{\left(\frac{n_1{d}_1+n_2{d}_2}{n_1+n_2}\right)}{w_{max}}\le \frac{w_{max}}{w_{sep}^r}.$$

$$w_2>\frac{2n_2{w}_1-n_1+\sqrt{n_1^2+4w_1(n_2^2{w}_1+N^2-N{n}_1-n_2^2)}}{2(N-n_1+n_2)}$$

$$DBCV{I}_2>DBCV{I}_1>0$$

$$⟺\frac{n_2}{N}(\frac{w_2}{w_1}-1)+(1-\frac{n_2}{N})(1-\frac{1}{w_2})$$

$$-\frac{n_1}{N}(1-\frac{1}{w_1})+(1-\frac{n_1}{N})(1-\frac{w_2}{w_1})>0$$

$$⟺w_2^2\underset{a}{\underbrace{(N+n_2-n_1)}}+w_2\underset{b}{\underbrace{(n_1-2n_2{w}_1)}}$$

$$+\underset{c<0}{\underbrace{(n_2-N)w_1}}>0.$$

$$DBCV{I}_1$$

$$DBCV{I}_2$$

$$=1-\frac{1}{w_2}+\frac{n_2}{N}(\underset{=A}{\underbrace{\frac{w_2}{w_1}+\frac{1}{w_2}-2}})$$

$$DBCV{I}_2>DBCV{I}_1$$

$$⟺-\frac{1}{w_2}+(\frac{1}{2}+\frac{1}{N})(\frac{w_2}{w_1}-2+\frac{1}{w_2})>-\frac{1}{w_1}-\frac{w_2}{w_1}$$

$$⟺(\frac{1}{2w_1}+\frac{1}{N{w}_1}+\frac{1}{2w_1})w_2-1-\frac{2}{N}+\frac{1}{2w_1}$$

$$+(-1+\frac{1}{2}+\frac{1}{N})\frac{1}{w_2}>0$$

$$⟺\underset{a>0}{\underbrace{(1+\frac{1}{N})}}{w}_2^2+w_2\underset{b<0}{\underbrace{(\frac{1}{2}-w_1(1+\frac{2}{N}))}}$$

$$+w_1\underset{c<0}{\underbrace{(\frac{1}{N}-\frac{1}{2})}}>0$$

$$w_1>w_2⟺w_1>\frac{N}{2(N+1)}[w_1(1+\frac{2}{N})-\frac{1}{2}+\sqrt{\Delta }]$$

$$⟺w_1+\frac{1}{2}>\sqrt{\Delta }$$

$$⟺\frac{4}{N}{w}_1^2(1+\frac{1}{N})+\frac{4}{N}{w}_1(-1-\frac{1}{N})<0$$

$$⟺w_1<1:ILLICIT$$

$$\mathrm{SEP}(C_i^l)\ge \min (\mathrm{SEP}(C_i),w_{max})\ge \mathrm{SEP}(C_i).$$

$$DBCV{I}_{aftercut}$$

$$\ge \sum _{j\ne i}\frac{n_j}{N}{V}_C(C_j)+\frac{n_i^l}{N}{V}_C(C_i)+\frac{n_i^r}{N}{V}_C(C_i)$$

$$=DBCV{I}_{beforecut}.$$

$$\frac{\left(\frac{n_1{d}_1+n_2{d}_2}{n_1+n_2}\right)}{w_{max}}\le \frac{w_{max}}{w_{sep}^r}.$$

$$DBCV{I}_K$$

$$DBCV{I}_{K+1}$$

$$+\frac{n_2}{N}\left(1-\frac{d_2}{w_{max}}\right)$$

Full text

1 Introduction

Clustering is one of the principal data mining tasks consisting in grouping related objects in an unsupervised manner. It is expected that objects belonging to the same cluster are more similar to each other than to objects belonging to different clusters. There exists a variety of algorithms performing this task. Methods like $k$-means [2], $k$-medians [3] or $k$-medoids [4] are useful unless the number and the shape of clusters are unknown which is unfortunately often the case in real-world applications. They are typically unable to find clusters with a nonconvex shape. Although DBSCAN [5] does not have these disadvantages, its resulting clustering still depends on the chosen parameter values.

One of the successful approaches relies on a graph representation of the data. Given a set of $N$ data points $\{x_{1},\dots,\ x_{N}\}$, a graph can be built based on the dissimilarity of data where points of the dataset are the vertices and weighted edges express distances between these objects. Besides, the dataset can be already a graph $\mathcal{G}$ modeling a network in many fields, such as bioinformatics - where gene-activation dependencies are described through a network - or social, computer, information, transportation network analysis. The clustering task consequently aims at detecting clusters as groups of nodes that are densely connected with each other and sparsely connected to vertices of other groups. In this context, Spectral Clustering (SC) [10] is a popular tool to recover clusters with particular structures for which classical $k$-means algorithm fails. When dealing with large scale datasets, a main bottleneck of the technique is to perform the partial eigendecomposition of the associated graph Laplacian matrix, though. Another inherent difficulty is to handle the huge number of nodes and edges of the induced dissimilarity graph: storing all edges can cost up to $O(N^{2})$ where $N$ is the number of nodes. Over the last decade, it has been established that the dynamic streaming model [6] associated with linear sketching techniques [7] - also suitable for distributed processing -, is a good way for tackling this last issue. In this paper, the addressed problem falls within a framework of storage limits allowing $O(N\operatorname{polylog}(N))$ space complexity but not $O(N^{2})$.

Contributions.

The new clustering algorithm DBMSTClu presented in this paper brings a solution to these issues: 1) detecting arbitrary-shaped data clusters, 2) with no parameter, 3) in a time linear to the number of points, 4) in a space-efficient manner by working on a limited number of linear measurements, a sketched version of the streamed dissimilarity graph $\mathcal{G}$. DBMSTClu returns indeed a partition of the $N$ points to cluster relying only on a Minimum Spanning Tree (MST) of the dissimilarity graph $\mathcal{G}$ taking $O(N)$ space. This MST can be space-efficiently approximatively retrieved in the dynamic semi-streaming model by handling $\mathcal{G}$ as a stream of edge weight updates sketched in only one pass over the data into a compact structure taking $O(N\operatorname{polylog}(N))$ space. DBMSTClu then automatically identifies the right number of nonconvex clusters by cutting suitable edges of the resulting approximate MST in $O(N)$ time.

The remaining of this paper is organized as follows. In §2 the related work about graph clustering and other space-efficient clustering algorithms is described. §3 gives fundamentals of the sketching technique that can be used to obtain an approximate MST of the dissimilarity graph. DBMSTClu (DB for Density-Based), the proposed MST-based algorithm for clustering is then explained in §4.1, its theoretical guarantees discussed in §4.2 and the implementation enabling its scalability detailed in §4.3. §5 presents the experimental results comparing the proposed clustering algorithm to other existing methods. Finally, §6 concludes the work and discusses future directions.

2 Related work

2.1 General graph clustering.

The approach of graph representation of the data has led to an extensive literature over graph clustering related to graph partitioning [8]. From the clustering methods point of view, DenGraph [11] proposes a graph version of DBSCAN which is able to deal with noise while work from [12] focuses on the problem of recovering clusters with considerably dissimilar sizes. Recent works include also approaches from convex optimization using low-rank decomposition of the adjacency matrix [13, 14, 15, 16]. These methods bring theoretical guarantees about the exact recovery of the ground truth clustering for the Stochastic Block Model [17, 18, 19] but demand to compute the eigendecomposition of a $N\times N$ matrix (resp. $O(N^{3})$ and $O(N^{2})$ for time and space complexity). Moreover they are restricted to unweighted graphs (weights in the work from [16] refer to likeliness of existence of an edge, not a distance between points).

2.2 MST-based graph clustering.

The MST is known to help recognizing clusters with arbitrary shapes. Clustering algorithms from this family identify clusters by performing suitable cuts among the MST edges. The first algorithm, called Standard Euclidean MST (SEMST) is from [20] and given a number of expected clusters, consists in deleting the heaviest edges from the Euclidean MST of the considered graph but this completely fails when the intra-cluster distance is lower than the inter-clusters one. For decades since MST-based clustering methods [21, 22] have been developed and can be classified into the group of density-based methods. MSDR [22] relies on the mean and the standard deviation of edge weights within clusters but will encourage clusters with points far from each other as soon as they are equally “far”. Moreover, it does not handle clusters with less than three points.

2.3 Space-efficient clustering algorithms.

Streaming $k$-means [24] is a one-pass streaming method for the $k$-means problem but still fails to detect clusters with nonconvex shapes since only the centroid point of each cluster is stored. This is not the case of CURE algorithm [25] which represents each cluster as a random sample of data points contained in it but this offline method has a prohibitive time complexity of $O(N^{2}\log(N))$ not suitable for large datasets. More time-efficient, CluStream [26] and DenStream [27] create microclusters based on local densities in an online fashion and aggregate them later to build bigger clusters in offline steps. Though, only DenStream can capture non-spherical clusters but needs parameters like DBSCAN from which it is inspired.

3 Context, notations and graph sketching preprocessing step

Consider a dataset with $N$ points. Either the underlying network already exists, or it is assumed that a dissimilarity graph $\mathcal{G}$ between points can be built where points of the dataset are the vertices and weighted edges express distances between these objects. For instance, this can be the Euclidean distance. In both cases, the graphs considered here should follow this definition:

Definition 3.1** (graph $\mathcal{G}=(V,\ E)$)**

A graph $\mathcal{G}=(V,\ E)$ consists in a set of nodes $V$ and a set of edges $E\subseteq V\times V$. The graph is undirected but weighted. The weight $w$ on an edge between node $i$ and $j$ - if this edge exists - corresponds to the normalized predefined distance between $i$ and $j$, s.t. $0<w\leq 1$. $|V|=N$ and $|E|=M$ stand resp. for the cardinality of sets $V$ and $E$. $E=\{e_{1},\ ...,\ e_{M}\}$ and for all edges $e_{i}$ a weight $w_{i}$ represents a distance between two vertices. In the sequel, $E(\mathcal{G})$ describes the set of edges of a graph $\mathcal{G}$. **

Freely of any parameter, DBMSTClu performs the nodes clustering from its unique input: an MST of $\mathcal{G}$. So an independent preprocessing is required for its recovery by any existing method. To respect our space restrictions though, the use of a graph sketching technique is motivated here: from the stream of its edge weights, a sketch of $\mathcal{G}$ is built and then an approximate MST is retrieved exclusively from it.

Streaming graph sketching and approximate MST recovery.

Processing data in the dynamic streaming model [6] for graph sketching implies: 1) The graph should be handled as a stream $s$ of edge weight updates: $s=(a_{1},...\ a_{j},...)$ where $a_{j}$ is the $j$-th update in the stream corresponding to the tuple $a_{j}=(i,\ w_{old,i},\ \Delta w_{i})$ with $i$ denoting the index of the edge to update, $w_{old,i}$ its previous weight and $\Delta w_{i}$ the update to perform. Thus, after reading $a_{j}$ in the stream, the $i$-th edge is assigned the new weight $w_{i}=w_{old,i}+\Delta w_{i}\geq 0$. 2) The method should make only one pass over this stream . 3) Edges can be both inserted or deleted (turnstile model), i.e. weights can be increased or decreased (but have always to be nonnegative). So weights change regularly, as in social networks where individuals can be friends for some time then not anymore.

The algorithm in [7] satisfies these conditions and is used here to produce in an online fashion a limited number of linear measurements summarizing edge weights of $\mathcal{G}$, as new data $a_{j}$ are read from the stream $s$ of edge weight updates. Its general principle is briefly described here. For a given small $\epsilon_{1}$, $\mathcal{G}$ is seen as a set of unweighted subgraphs $\mathcal{G}_{k}$ containing all the edges with weight lower than $(1+\epsilon_{1})^{k}$, hence $\mathcal{G}_{k}\subset\mathcal{G}_{k+1}$. The $\mathcal{G}_{k}$ are embodied as $N$ virtual vectors $v^{(i)}\in\{-1,0,1\}^{M}$ for $i\in[N]$111In the sequel, for a given integer $a$, $[a]=\{1,\ldots,a\}$. expressing for each node the belonging to an existing edge: for $j\in[M]$, $v^{(i)}_{j}$ equals to [math] if node $i$ is not in $e_{j}$, $1$ (resp. $-1$) if $e_{j}$ exists and $i$ is its left (resp. right) node. All $v^{(i)}$ are described at $L$ different “levels”, i.e. $L$ virtual copies of the true vectors are made with some entries randomly set to zero s.t. the $v^{(i),l}$ get sparser as the corresponding level $l\in[L]$ increases. The $v^{(i),l}$ for each level are explicitly coded in memory by three counters: $\phi=\sum_{j=1}^{M}v^{(i),l}_{j}$; $\iota=\sum_{j=1}^{M}j\ v^{(i),l}_{j}$; $\tau=\sum_{j=1}^{M}v^{(i),l}_{j}\ z^{j}\mod p$, with $p$ a suitably large prime and $z\in\mathbb{Z}_{p}$. The resulting compact data structure further named a sketch enables to draw almost uniformly at random a nonzero weighted edge among $\mathcal{G}_{k}$ at any time among the levels vectors $v^{(i),l}$ which are $1$-sparse (with exactly one nonzero coefficient) thanks to $\ell_{0}$-sampling [28]:

Definition 3.2** ($\ell_{0}$-sampling)**

An $(\epsilon,\delta)$ $\ell_{0}$-sampler for a nonzero vector $x\in\mathbb{R}^{n}$ fails with a probability at most $\delta$ or returns some $i\in[n]$ with probability $(1\pm\epsilon)\frac{1}{|\operatorname{supp}x|}$ where $\operatorname{supp}x=\{i\in[n]\ |\ x_{i}\neq 0\}$. **

The sketch requires $O(N\log^{3}(N))$ space. It follows that the sketching is technically semi-streamed but in practice only one pass over the data is needed and the space cost is significantly lower than the theoretical $O(N^{2})$ bound. The time cost for each update of the sketch is $\operatorname{polylog}(N)$. The authors from [7] also proposed an algorithm to compute in a single-pass the approximate weight $\tilde{W}$ of an MST $\mathcal{T}$ - the sum of all its edge weights - by appropriate samplings from the sketch in $O(N\operatorname{polylog}(N))$ time. They show that $W\leq\tilde{W}\leq(1+\epsilon_{1})\ W$ where $W$ stands for the true weight and $\tilde{W}=N-(1+\epsilon_{1})^{r+1}\ cc(\mathcal{G}_{r})+\sum^{r}_{k=0}\lambda_{k}\ cc(\mathcal{G}_{k})$ with $\lambda_{k}=(1+\epsilon_{1})^{k+1}-(1+\epsilon_{1})^{i}$, $r=\lceil\log_{1+\epsilon_{1}}(w_{max})\rceil$ s.t. $w_{max}$ is the maximal weight of $\mathcal{G}$ and $cc$ denotes the number of connected components of the graph in parameter. Here an extended method is applied for obtaining rather an approximate MST - and not simply its weight - by registering edges as they are sampled. Referring to the proof of Lemma 3.4 in [7], the approach is simply justified by applying Kruskal’s algorithm where edges with lower weights are first sampled222We would like to thank Mario Lucic for the fruitful private conversation and for coauthoring with Krzysztof Choromanski the MSE sketching extension during his internship at Google..

Note that the term MST is kept in the whole paper for the sake of simplicity, but the sketching technique and so does our algorithm enables to recover a Minimum Spanning Forest if the initial graph is disconnected.

4 The proposed MST-based graph clustering method: DBMSTClu

4.1 Principle.

Let us consider a dataset with $N$ points. After the sketching phase, an approximate MST further named $\mathcal{T}$ has been obtained with $N-1$ edges s.t. $\forall i\in[N-1],\ 0<w_{i}\leq 1$. Our density-based clustering method DBMSTClu exclusively relies on this object by performing some cuts among the edges of the tree s.t. $K-1$ cuts result in $K$ clusters. Note that independently of the technique used to obtain an MST (the sketching method is just a possible one), the space complexity of the algorithm is $O(N)$ which is better than the $O(N^{2})$ of Spectral Clustering (SC). The time complexity of DBMSTClu is $O(NK)$ which is clearly less than the $O(N^{3})$ one implied by SC. After a cut, obtained clusters can be seen as subtrees of the initial $\mathcal{T}$ and the analysis of their qualities is only based on edges contained in those subtrees. In the sequel, all clusters $C_{i}$, $i\in[K]$, are assimilated to their associated subtree of $\mathcal{T}$ and for instance the maximal edge of a cluster will refer to the edge with the maximum weight from the subtree associated to this cluster.

Our algorithm is a parameter-free divisive top-down procedure: it starts from one cluster containing the whole dataset and at each iteration, a cut which maximizes some criterion is performed. The criterion for identifying the best cut to do (if any should be made) at a given stage is a measure of the validity of the resulting clustering partition. This is a function of two positive quantities defined below: Dispersion and Separation of one cluster. The quality of a given cluster is then measured from Dispersion and Separation while the quality of the clustering partition results from the weighted average of all cluster validity indices. Finally, all those measures are based on the value of edge weights and the two latter ones lie between $-1$ and $1$.

Definition 4.1** (Cluster Dispersion)**

The Dispersion of a cluster $C_{i}$ ($\operatorname{DISP}$) is defined as the maximum edge weight of $C_{i}$. If the cluster is a singleton (i.e. contains only one node), the associated Dispersion is set to [math]. More formally:

[TABLE]

Definition 4.2** (Cluster Separation)**

The Separation of a cluster $C_{i}$ ($\operatorname{SEP}$) is defined as the minimum distance between the nodes of $C_{i}$ and the ones of all other clusters $C_{j},j\neq i,1\leq i,j\leq K,K\neq 1$ where $K$ is the total number of clusters. In practice, it corresponds to the minimum weight among all already cut edges from $\mathcal{T}$ comprising a node from $C_{i}$. If $K=1$, the Separation is set to $1$. More formally, with $Cuts(C_{i})$ denoting cut edges incident to $C_{i}$,

[TABLE]

Fig. 1 sums up the introduced definitions. The higher the Separation, the farther is the cluster separated from the other clusters, while low values suggest that the cluster is close to the nearest one.

Definition 4.3** (Validity Index of a Cluster)**

The Validity Index of a cluster $C_{i}$ is defined as:

[TABLE]

The Validity Index of a Cluster (illustration in Fig. 2) is defined s.t. $-1\leq V_{C}(C_{i})\leq 1$ where $1$ stands for the best validity index and $-1$ for the worst one. No division by zero (i.e. $\max(\operatorname{DISP}(C_{i}),\operatorname{SEP}(C_{i}))=0$) happens because Separation is always strictly positive. When Dispersion is higher than Separation, $-1<V_{C}(C_{i})<0$. Conversely, when Separation is higher than Dispersion, $0<V_{C}(C_{i})<1$. So our clustering algorithm will naturally encourage clusters with a higher Separation over those with a higher Dispersion.

Definition 4.4

(Validity Index of a Clustering Partition)* *The Density-Based Validity Index of a Clustering partition $\Pi=\{C_{i}\},1\leq i\leq K$, $\operatorname{DBCVI}(\Pi)$ is defined as the weighted average of the Validity Indices of all clusters in the partition where $N$ is the number of points in the dataset.

[TABLE]

The Validity Index of Clustering lies also between $-1$ and $1$ where $1$ stands for an optimal density-based clustering partition while $-1$ stands for the worst one.

Our defined quantities are significantly distinct from the separation and sparseness defined in [29]. Indeed, firstly, their quantities are not well defined for special cases when clusters have less than four nodes or a partition containing a lonely cluster. Secondly, the way they differentiate internal and external nodes or edges does not properly recover easy clusters like convex blobs. Moreover, our DBCVI differs from the Silhouette Coefficient [30]. It does not perform well with nonconvex-shaped clusters and although this is based on close concepts like tightness and also separation, the global coefficient is based on the average values of Silhouette coefficients of each point, while our computation of DBCVI begins at the cluster level.

DBMSTClu is summarized in Algorithm 1. It starts from a partition with one cluster containing the whole dataset whereas the associated initial DBCVI is set to the worst possible value: $-1$. As long as there exists a cut which makes the DBCVI greater from (or equal to) the one of the current partition, a cut is greedily chosen by maximizing the obtained DBCVI among all the possible cuts. When no direct improvement is possible, the algorithm stops. It is guaranteed that the cut edge locally maximizes the DBCVI at each iteration since by construction, the algorithm will try each possible cut. In practice, the algorithm stops after a reasonable number of cuts, getting trapped in a local maximum corresponding to a meaningful cluster partition. This prevents from obtaining a partition where all points are in singleton clusters. Indeed, such a result ($K=N$) is not desirable, although it is optimal in the sense of the DBCVI, since in this case, $\forall i\in[K]$, $\operatorname{DISP}(C_{i})=0$ and $V_{C}(C_{i})=1$. Moreover, the non-parametric characteristic helps achieving stable partitions. In Algorithm 1, $\operatorname{evaluateCut}(.)$ computes the DBCVI when the cut in parameter is applied to $\mathcal{T}$.

4.2 Quality of clusters.

An analysis of the algorithm and the quality of the recovered clusters is now given. The main results are: 1) DBMSTClu differs significantly from the naive approach of SEMST by preferring cuts which do not necessarily correspond to the heaviest edge (Prop. 4.1 and 4.2). 2) As long as the current partition contains at least one cluster with a negative validity index, DBMSTClu will find a cut improving the global index (Prop. 4.3). 3) Conditions are given to determine in advance if and which cut will be performed in a cluster with a positive validity index (Prop. 4.4 and 4.5 ). All are completely independent of the sketching phase. Prop. 4.1 and 4.2 rely on the two basic lemmas regarding the first cut in the MST:

Lemma 4.1

Highest weighted edge case* Let $\mathcal{T}$ be an MST of the dissimilarity data graph. If the first cut from $E(\mathcal{T})$ made by DBMSTClu is the heaviest edge, then resulting DBCVI is nonnegative.*

• Proof.

For the first cut, both separations of obtained clusters $C_{1}$ and $C_{2}$ are equal to the weight of the considered edge for cut. Here, this is the one with the highest weight. Thus, for $i=1,2$, $\operatorname{DISP}(C_{i})\leq\operatorname{SEP}(C_{i})\small{\implies}V_{C}(C_{i})\geq 0$. Finally, the DBCVI of the partition, as a convex sum of two nonnegative quantities, is clearly nonnegative.

Lemma 4.2

Lowest weighted edge case* Let $\mathcal{T}$ be an MST of the dissimilarity data graph. If the first cut from $E(\mathcal{T})$ done by DBMSTClu is the one with the lowest weight, then resulting DBCVI is negative or zero.*

• Proof.

Same reasoning in the opposite case s.t. $\operatorname{SEP}(C_{i})-\operatorname{DISP}(C_{i})\leq 0$ for $i\in\{1,2\}$.

Proposition 4.1

When the first cut is not the heaviest* Let $\mathcal{T}$ be an MST of the dissimilarity data graph with $N$ nodes. Let us consider this specific case: all edges have a weight equal to $w$ except two edges $e_{1}$ and $e_{2}$ resp. with weight $w_{1}$ and $w_{2}$ s.t. $w_{1}>w_{2}>w>0$. DBMSTClu does not cut any edge with weight $w$ and cuts $e_{2}$ instead of $e_{1}$ as a first cut iff:*

[TABLE]

where $n_{1}$ (resp. $n_{2}$) is the number of nodes in the first cluster resulting from the cut of $e_{1}$ (resp. $e_{2}$). Otherwise, $e_{1}$ gets cut.

• Proof.

See supplementary material.

This proposition emphasizes that the algorithm is cleverer than simply cutting the heaviest edge first. Indeed, although $w_{2}<w_{1}$, cutting $e_{2}$ could be preferred over $e_{1}$. Moreover, no edge with weight $w$ can get cut at the first iteration as they have the minimal weight in the tree. Indeed it really happens since an approximate MST with discrete rounded weights is used when sketching is applied.

Proposition 4.2

First cut on the heaviest edge in the middle* Let $\mathcal{T}$ be an MST of the dissimilarity data graph with $N$ nodes. Let us consider this specific case: all edges have a weight equal to $w$ except two edges $e_{1}$ and $e_{2}$ resp. with weight $w_{1}$ and $w_{2}$ s.t. $w_{1}>w_{2}>w>0$. Denote $n_{1}$ (resp. $n_{2}$) the number of nodes in the first cluster resulting from the cut of $e_{1}$ (resp. $e_{2}$). In the particular case where edge $e_{1}$ with maximal weight $w_{1}$ stands between two subtrees with the same number of points, i.e. $n_{1}=N/2$, $e_{1}$ is always preferred over $e_{2}$ as the first optimal cut.*

• Proof.

See supplementary material.

Remark 4.1

Let us consider the MST in Fig. 3 with $N=8$, $w_{1}=1$, $w_{2}=w_{3}=1-\epsilon$, and other weights set to $\epsilon$. Clearly, it is not always preferred to cut $e_{1}$ in the middle since for $\epsilon=0.1$, $DBCVI_{2}\approx 0.27>DBCVI_{1}=\epsilon=0.1$. So, it is a counter-example to a possible generalization of Prop. 4.2 where there would be more than three possible distinct weights in $\mathcal{T}$.

These last propositions hold for every iteration in the algorithm.

Proposition 4.3

Fate of negative $V_{C}$ cluster*

Let $K=t+1$ be the number of clusters in the clustering partition at iteration $t$. If for some $i\in[K],\ V_{C}(C_{i})<0$, then DBMSTClu will cut an edge at this stage.*

• Proof.

See supplementary material.

Thus, at the end of the clustering algorithm, each cluster will have a nonnegative $V_{C}$ so at each step of the algorithm, this bound holds for the final DBCVI: $DBCVI\geq\sum_{i=1}^{K}\frac{|C_{i}|}{N}\max(V_{C}(C_{i}),0)$.

Proposition 4.4

Fate of positive $V_{C}$ cluster I* Let $\mathcal{T}$ be an MST of the dissimilarity data graph and $C$ a cluster s.t. $V_{C}(C)>0$ and $\operatorname{SEP}(C)=s$. DBMSTClu does not cut an edge $e$ of $C$ with weight $w<s$ if both resulting clusters have at least one edge with weight greater than $w$.*

• Proof.

See supplementary material.

Proposition 4.5

Fate of positive $V_{C}$ cluster II*

Consider a partition with $K$ clusters s.t. some cluster $C_{i}$, $i\in[K]$ with $V_{C}(C_{i})>0$ is in the setting of Fig. 10 i.e. cutting the heaviest edge $e$ with weight $w_{max}$ results in two clusters: the left (resp. right) cluster $C_{i}^{l}$ (resp. $C_{i}^{r}$) with $n_{1}$ points (resp. $n_{2}$) s.t. $\operatorname{DISP}(C_{i}^{l})=d_{1}$, $\operatorname{SEP}(C_{i}^{l})=w^{l}_{sep}$, $\operatorname{DISP}(C_{i}^{r})=d_{2}$ and $\operatorname{SEP}(C_{i}^{r})=w^{r}_{sep}$. Assuming w.l.o.g. $w^{l}_{sep}>w^{r}_{sep}$, cutting edge $e$ improves the DBCVI iff:*

[TABLE]

• Proof.

See supplementary material.

4.3 Implementation for linear time and space complexities.

Algorithm 1 previously described could lead to a naive implementation. We now briefly explain how to make the implementation efficient in order to achieve the linear time and space complexities in $N$ (code on https://github.com/annemorvan/DBMSTClu/). The principle is based on two tricks.

1. As observed in §4.2: for a performed cut in cluster $C_{i}$, $V_{C}(C_{j})$ for any $j\neq i$ remain unchanged. Hence, if $V_{C}(C_{j})$ are stored for each untouched cluster after a given cut, only the edges of $C_{l}$ and $C_{r}$ resp. the left and right clusters induced by $e$’s cut need to be evaluated again to determine the DBCVI in case of cut. Thus the number of operations to find the optimal cut decreases drastically over time as the clusters become smaller through the cuts.
2. However finding the first cut already costs $O(N)$ time hence paying this price for each cut evaluation would lead to $O(N^{2})$ operations. Fortunately, this can be avoided as $\operatorname{SEP}$ and $\operatorname{DISP}$ exhibit some recurrence relationship in $\mathcal{T}$: when knowing these values for a given cut, we can deduce the value for a neighboring cut (cf. Fig. 5). To determine the first cut, $\mathcal{T}$ should be hence completely crossed following the iterative version of the Depth-First Search. The difficulty though is that the recursive relationship between the quantities to update is directional: left and right w.r.t. the edge to cut. So we develop here double Depth-First search (see principle in Algorithm 2): from any given edge of $\mathcal{T}$, edges left and right are all visited consecutively with a Depth-First search, and $\operatorname{SEP}$ and $\operatorname{DISP}$ are updated recursively thanks to a carefully defined order in the priority queue of edges to handle.

5 Experiments

Tight lower and upper bounds are given in §3 for the weight of the approximate MST retrieved by sketching. First experiments in §5.1 show that the clustering results do not suffer from the use of an approximate MST instead of a real one. Experiments from §5.2 prove then the scalability of DBMSTClu for large values of $N$.

5.1 Safety of the sketching.

The results of DBMSTClu are first compared with DBSCAN [5] because it can compete with DBMSTClu as 1) nonconvex-shaped clusters are recognized, 2) it does not require explicitly the number of expected clusters, 3) it is both time and space-efficient (resp. $O(N\log N)$ and $O(N)$). Then, the results of another MST-based algorithm are shown for comparison. The latter called Standard Euclidean MST (SEMST) [20] cuts the $K-1$ heaviest edges of a standard Euclidean MST given a targeted number of clusters $K$. For DBMSTClu, the dissimilarity graph is built from computing the Euclidean distance between data points and passed into the sketch phase to produce an approximate version of the exact MST.

Synthetic datasets. Experiments were performed on two classic synthetic datasets from the Euclidean space: noisy circles and noisy moons. Each dataset contains $1000$ data points in $20$ dimensions: the first two dimensions are randomly drawn from predefined 2D-clusters, as shown in Fig. 6 and 7, while the other $18$ dimensions are random Gaussian noise.

Real dataset. DBMSTClu performances are also measured on the mushroom dataset (https://archive.ics.uci.edu/ml/datasets/mushroom). It contains $8124$ records of $22$ categorical attributes corresponding to $23$ species of gilled mushrooms in the Agaricus and Lepiota family. $118$ binary attributes are created from the $22$ categorical ones, then the complete graph (about $33$ millions of edges) is built by computing the normalized Hamming distance (i.e. the number of distinct bits) between points.

Results. Fig. 6, 7 and 8 show the results for all previously defined datasets and aforementioned methods. The synthetic datasets were projected onto 2D spaces for visualization purposes. They were produced with a noise level such that SEMST fails and DBSCAN does not perform well without parameters optimization. In particular, for DBSCAN all the cross points correspond to noise. With the concentric circles, SEMST does not cut on the consistent edges, hence leads to an isolated singleton cluster. DBSCAN classifies the same point plus a near one as noise while recovering the two circles well. Finally, DBMSTClu finds the two main clusters and also creates five singleton clusters which can be legitimately considered as noise as well. With noisy moons, while DBSCAN considers three outliers, DBMSTClu detects the same as singleton clusters. As theoretically proved above, experiments emphasize the fact that our algorithm is more subtle than simply cutting the heaviest edges as the failure of SEMST shows. Moreover our algorithm exhibits an ability to detect outliers, which could be labeled as noise in a postprocessing phase. Another decisive advantage of our algorithm is the absence of any required parameters, contrarily to DBSCAN. For the mushroom dataset, if suitable parameters are given to SEMST and DBSCAN, the right $23$ clusters get found while DBMSTClu retrieves them without tuning any parameter. Quantitative results for the synthetic datasets are shown in Table 3: the achieved silhouette coefficient (between $-1$ and $1$), Adjusted Rand Index (ARI) (between [math] and $1$) and DBCVI. For all the indices, the higher, the better. Further analysis can be read in supplementary material. For the mushroom dataset, the corresponding DBCVI and silhouette coefficient are resp. $0.75$ and $0.47$.

5.2 Scalability of the clustering.

For mushroom dataset, DBMSTClu’s execution time (avg. on $5$ runs) is $3.36$s while DBSCAN requires $9.00$s. This gives a first overview of its ability to deal with high number of clusters. Further experiments on execution time were conducted on large-scale random weighted graphs generated from the Stochastic Block Model with varying equal-sized number of clusters $K$ and $N$. The scalability of DBMSTClu is shown in Fig. 9 and Table 2 by exhibiting the linear time complexity in $N$. Graphs with $1$M of nodes and $100$ clusters were easily clustered. In Table 2, the row with the execution time ratio between $K=100$ and $K=5$ illustrates the first trick from §4.3 as the observed time ratio is around $2/3$ of the theoretical one $100/5=20$.

6 Conclusion

In this paper we introduced DBMSTClu a novel space-efficient Density-Based Clustering algorithm which only relies on a Minimum Spanning Tree (MST) of the dissimilarity graph $\mathcal{G}$: the spatial and time costs are resp. $O(N)$ and $O(NK)$ with $N$ the number of data points and $K$ the number of clusters. This enables to deal easily with graphs of million of nodes. Moreover, DBMSTClu is non-parametric, unlike most existing clustering methods: it automatically determines the right number of nonconvex clusters. Although the approach is fundamentally independent from the sketching phase, its robustness has been assessed by using as input an approximate MST of the sketched $\mathcal{G}$ rather than an exact one. The graph sketch is computed dynamically on the fly as new edge weight updates are read in only one pass over the data. This brings a space-efficient solution for finding an MST of $\mathcal{G}$ when the $O(N^{2})$ edges cannot fit in memory. Hence, our algorithm adapts to the semi-streaming setting with $O(N\operatorname{polylog}N)$ space. Our approach shows promising results, as evidenced by the experimental part regarding time and space scalability and clustering performance even with sketching. Further work would consist in using this algorithm in privacy issues, as the lost information when sketching might ensure data privacy. Moreover, as it is already the case for the graph sketching, we could look for adapting both the MST recovery and DBMSTClu to the fully online setting, i.e. to be able to modify dynamically current MST and clustering partition as a new edge weight update from the stream is seen.

7 Proofs.

7.1 Proof of Prop. 4.1.

Proposition 4.1. When the first cut is not the heaviest Let $\mathcal{T}$ be an MST of the dissimilarity data graph with $N$ nodes. Let us consider this specific case: all edges have a weight equal to $w$ except two edges $e_{1}$ and $e_{2}$ resp. with weight $w_{1}$ and $w_{2}$ s.t. $w_{1}>w_{2}>w>0$. DBMSTClu does not cut any edge with weight $w$ and cuts $e_{2}$ instead of $e_{1}$ as a first cut iff:

[TABLE]

where $n_{1}$ (resp. $n_{2}$) is the number of nodes in the first cluster resulting from the cut of $e_{1}$ (resp. $e_{2}$). Otherwise, $e_{1}$ gets cut.

• Proof.

Let $DBCVI_{1}$ (resp. $DBCVI_{2}$) be the DBCVI after cut of $e_{1}$ (resp. $e_{2}$). As $w$ (resp. $w_{1}$) is the minimum (resp. maximal) weight, the algorithm does not cut $e$ since the resulting DBCVI would be negative (cf. Lemma 4.2) while $DBCVI_{1}$ is guaranteed to be positive (cf. Lemma 4.1). So, the choice will be between $e_{1}$ and $e_{2}$ but $e_{2}$ gets cut iff $DBCVI_{2}>DBCVI_{1}$. $DBCVI_{1}$ and $DBCVI_{2}$ expressions are simplified w.l.o.g. by scaling the weights by $w$ s.t. $w\leftarrow 1$, $w_{1}\leftarrow w_{1}/w$, $w_{2}\leftarrow w_{2}/w$, hence $w_{1}>w_{2}>1$. Then,

[TABLE]

Clearly, $\Delta=b^{2}-4ac$ is positive and $c/a$ is negative. But $w_{2}>0$, then $w_{2}>\frac{-b+\sqrt{b^{2}-4ac}}{2a}$ which gives the final result after some simplifications.

7.2 Proof of Prop. 4.2.

Proposition 4.2. First cut on the heaviest edge in the middle Let $\mathcal{T}$ be an MST of the dissimilarity data graph with $N$ nodes. Let us consider this specific case: all edges have a weight equal to $w$ except two edges $e_{1}$ and $e_{2}$ resp. with weight $w_{1}$ and $w_{2}$ s.t. $w_{1}>w_{2}>w>0$. Denote $n_{1}$ (resp. $n_{2}$) the number of nodes in the first cluster resulting from the cut of $e_{1}$ (resp. $e_{2}$). In the particular case where edge $e_{1}$ with maximal weight $w_{1}$ stands between two subtrees with the same number of points, i.e. $n_{1}=N/2$, $e_{1}$ is always preferred over $e_{2}$ as the first optimal cut.

• Proof.

A reductio ad absurdum is made by showing that cutting edge $e_{2}$ i.e. $DBCVI_{2}>DBCVI_{1}$ leads to the contradiction $w_{1}/w<1$. With the scaling process from Prop. 4.1’proof:

[TABLE]

There is $w_{2}>w=1$, so $\frac{1}{w_{2}}<1$. Besides $w_{2}<w_{1}$ so $\frac{w_{2}}{w_{1}}<1$ thus, $A<0$. Let now consider w.l.o.g. that edge $e_{2}$ is on the ”right side” (right cluster/subtree) of $e_{1}$ (similar proof if $e_{2}$ is on the left side of $e_{1}$). Hence, it is clear that for maximizing $DBCVI_{2}$ as a function of $n_{2}$, we need $n_{2}=n_{1}+1$. Then,

[TABLE]

As $c/a<0$ and $w_{2}>0$, $w_{2}>\frac{N}{2(N+1)}\ [\ w_{1}(1+\frac{2}{N})-\frac{1}{2}+\sqrt{\Delta}\ ]$ with $\Delta=(w_{1}(1+\frac{2}{N})-\frac{1}{2})^{2}+4(1+\frac{1}{N})(\frac{1}{2}-\frac{1}{N})w_{1}$. This inequality is incompatible with $w_{1}>w_{2}$ since:

[TABLE]

Indeed, after the scaling process, $w_{1}<1=w$ is not possible since by hypothesis, $w_{1}>w$. Finally, it is not allowed to cut $e_{2}$, the only remaining possible edge to cut is $e_{1}$.

7.3 Proof of Prop. 4.3.

Proposition 4.3. Fate of negative $V_{C}$ cluster Let $K=t+1$ be the number of clusters in the clustering partition at iteration $t$. If for some $i\in[K],\ V_{C}(C_{i})<0$, then DBMSTClu will cut an edge at this stage.

• Proof.

Let $i\in[K]$ s.t. $V_{C}(C_{i})<0$ i.e. $\operatorname{SEP}(C_{i})<\operatorname{DISP}(C_{i})$. We denote $w_{sep}^{l}$ the minimal weight outing cluster $C_{i}$ and $w_{max}$ the maximal weight in subtree $S_{i}$ of $C_{i}$ i.e. $\operatorname{SEP}(C_{i})\overset{def}{=}w_{sep}^{l}$ and $\operatorname{DISP}(C_{i})\overset{def}{=}w_{max}$. Hence, $w_{sep}^{l}<w_{max}$. By cutting the cluster $C_{i}$ on the edge with weight $w_{max}$, we define $C_{i}^{l}$ and $C_{i}^{r}$ resp. the left and right resulting clusters.

Let us look at $V_{C}(C_{i}^{l})$. If $\operatorname{SEP}(C_{i}^{l})\geq\operatorname{DISP}(C_{i}^{l})$ then $V_{C}(C_{i}^{l})\geq 0\geq V_{C}(C_{i})$ else $V_{C}(C_{i}^{l})=\frac{\operatorname{SEP}(C_{i}^{l})}{\operatorname{DISP}(C_{i}^{l})}-1$. The definition of the Separation as a minimum and our cut imply that

[TABLE]

Also the definition of the Dispersion as a maximum implies that $\operatorname{DISP}(C_{i}^{l})\leq\operatorname{DISP}(C_{i})$. Hence we get that $\frac{\operatorname{SEP}(C_{i}^{l})}{\operatorname{DISP}(C_{i}^{l})}-1\geq\frac{\operatorname{SEP}(C_{i})}{\operatorname{DISP}(C_{i})}-1$ i.e. $V_{C}(C_{i}^{l})\geq V_{C}(C_{i})$ in this case too. The same reasoning holds for $C_{i}^{r}$ showing that $V_{C}(C_{i}^{r})\geq V_{C}(C_{i})$. Finally,

[TABLE]

Hence cutting the edge with maximal weight in $C_{i}$ improves the resulting DBCVI.

7.4 Proof of Prop. 4.4.

Proposition 4.4. Fate of positive $V_{C}$ cluster I Let $\mathcal{T}$ be an MST of the dissimilarity data graph and $C$ a cluster s.t. $V_{C}(C)>0$ and $\operatorname{SEP}(C)=s$. DBMSTClu does not cut an edge $e$ of $C$ with weight $w<s$ if both resulting clusters have at least one edge with weight greater than $w$.

• Proof.

Let us consider clusters $C_{1}$ and $C_{2}$ resulting from the cut of edge $e$. Assume that in the associated subtree of $C_{1}$ (resp. $C_{2}$), there is an edge $e_{1}$ (resp. $e_{2}$) with a weight $w_{1}$ (resp. $w_{2}$) higher than $w$ s.t. without loss of generality, $w_{1}>w_{2}$. Since $V_{C}(C)>0$, $s>w_{1}>w_{2}>w$. But cutting edge $e$ implies that for $i\in\{1,2\}$, $\operatorname{DISP}(C_{i})>\operatorname{SEP}(C_{i})=w$, and thus $V_{C}(C_{i})<0$. Cutting edge $e$ would therefore mean to replace a cluster $C$ s.t. $V_{C}(C)>0$ by two clusters s.t. for $i\in\{1,2\}$, $V_{C}(C_{i})<0$ which obviously decreases the current DBCVI. Thus, $e$ does not get cut at this step of the algorithm.

7.5 Proof of Prop. 4.5.

Proposition 4.5. Fate of positive $V_{C}$ cluster II Consider a partition with $K$ clusters s.t. some cluster $C_{i}$, $i\in[K]$ with $V_{C}(C_{i})>0$ is in the setting of Fig. 10 i.e. cutting the heaviest edge $e$ with weight $w_{max}$ results in two clusters: the left (resp. right) cluster $C_{i}^{l}$ (resp. $C_{i}^{r}$) with $n_{1}$ points (resp. $n_{2}$) s.t. $\operatorname{DISP}(C_{i}^{l})=d_{1}$, $\operatorname{SEP}(C_{i}^{l})=w^{l}_{sep}$, $\operatorname{DISP}(C_{i}^{r})=d_{2}$ and $\operatorname{SEP}(C_{i}^{r})=w^{r}_{sep}$. Assuming w.l.o.g. $w^{l}_{sep}>w^{r}_{sep}$, cutting edge $e$ improves the DBCVI iff:

[TABLE]

• Proof.

As $V_{C}(C_{i})>0$, there is $\operatorname{SEP}(C_{i})=w^{r}_{sep}>w_{max}$. Then, the DBCVI before ($K$ clusters) and after cut of $w_{max}$ ($K+1$ clusters) are:

[TABLE]

DBMSTClu cuts $w_{max}$ iff $DBCVI_{K+1}\geq DBCVI_{K}$. So the result after simplification.

8 Complements on experiments.

Experiments were conducted using Python and scikit-learn library [1] on a single-thread process on an intel processor based node.

8.1 Safety of the sketching.

Fig. 11 shows another result on a synthetic dataset: three blobs generated from three Gaussian distributions. With the three blobs, each method SEMST, DBSCAN and DBMSTClu performs well: they all manage to retrieve three clusters.

Quantitative results for the three synthetic datasets are shown in Table 3: the achieved silhouette coefficient, Adjusted Rand Index (ARI) and DBCVI. For all the indices, the higher, the better. Silhouette coefficient (between $-1$ and $1$) is used to measure a clustering partition without any external information. For DBSCAN it is computed by considering noise points as singletons. We see that this measure is not very suitable for nonconvex clusters like noisy circles or moons. The ARI (between [math] and $1$) measures the similarity between the experimental clustering partition and the known groundtruth. DBSCAN and DBMSTClu give similar almost optimal results. Finally, the obtained DBCVIs are consistent, since the best ones are reached for DBMSTClu.