Fast communication-efficient spectral clustering over distributed data

TL;DR

This paper introduces a communication-efficient spectral clustering framework for distributed data that achieves near-accurate results with significant speedup and privacy benefits, especially when data are evenly distributed across sites.

Contribution

It presents a novel distributed spectral clustering method that minimizes communication, maintains accuracy, and enhances privacy, addressing limitations of existing algorithms assuming centralized data.

Findings

Achieves about 2x speedup in distributed spectral clustering.

Maintains negligible accuracy loss compared to centralized methods.

Addresses privacy concerns by transmitting data in non-original form.

Abstract

The last decades have seen a surge of interests in distributed computing thanks to advances in clustered computing and big data technology. Existing distributed algorithms typically assume {\it all the data are already in one place}, and divide the data and conquer on multiple machines. However, it is increasingly often that the data are located at a number of distributed sites, and one wishes to compute over all the data with low communication overhead. For spectral clustering, we propose a novel framework that enables its computation over such distributed data, with "minimal" communications while a major speedup in computation. The loss in accuracy is negligible compared to the non-distributed setting. Our approach allows local parallel computing at where the data are located, thus turns the distributed nature of the data into a blessing; the speedup is most substantial when the data are evenly distributed across sites. Experiments on synthetic and large UC Irvine datasets show almost no loss in accuracy with our approach while about 2x speedup under various settings with two distributed sites. As the transmitted data need not be in their original form, our framework readily addresses the privacy concern for data sharing in distributed computing.

Tables

Table 1. Table 1: A summary of UC Irvine data used in the experiments.

Data set	# Features	# instances	# classes
Connect-4	42	67,557	3
SkinSeg	3	245,057	2
USCI	37	285,779	2
Cover Type	54	568,772	5
HT Sensor	11	928,991	3
Poker Hand	10	1,000,000	3
Gas Sensor	18	8,386,765	2
HEPMASS	28	10,500,000	2

Table 2. Table 2: A summary of simulation settings. Here D 1 , D 2 , D 3 subscript 𝐷 1 subscript 𝐷 2 subscript 𝐷 3 D_{1},D_{2},D_{3} are 3 distributed simulation scenarios considered in this work.

Data set	${𝑫}_{\mathrm{𝟏}}$		${𝑫}_{\mathrm{𝟐}}$		${𝑫}_{\mathrm{𝟑}}$
USCI	${𝒞}_1$	${𝒞}_2$	0.7${𝒞}_1$+0.3${𝒞}_2$	0.3${𝒞}_1$+0.7${𝒞}_2$	50%	50%
SkinSeg
Gas Sensor
HEPMASS
Connect-4	${𝒞}_1$	${𝒞}_2$+${𝒞}_3$	0.5${𝒞}_1$+${𝒞}_2$	0.5${𝒞}_1$+${𝒞}_3$	50%	50%
HT Sensor
Poker Hand
Cover type	${𝒞}_2$	${𝒞}_1$+${𝒞}_{3-5}$	0.7${𝒞}_1$+0.3${𝒞}_2$	0.3${𝒞}_1$+0.7${𝒞}_2$	50%	50%
			+${𝒞}_{3-5}$

Table 3. Table 3: Clustering accuracy and running time on UC Irvine data under 3 simulation settings, D 1 , D 2 subscript 𝐷 1 subscript 𝐷 2 D_{1},D_{2} and D 3 subscript 𝐷 3 D_{3} , when K-means clustering is used as the DML. The time is in seconds. There are two distributed sites under all of D 1 , D 2 subscript 𝐷 1 subscript 𝐷 2 D_{1},D_{2} and D 3 subscript 𝐷 3 D_{3} .

Data set	Non-distributed	${𝑫}_{\mathrm{𝟏}}$	${𝑫}_{\mathrm{𝟐}}$	${𝑫}_{\mathrm{𝟑}}$
Connect-4	0.6569	0.6576	0.6569	0.6569
	36	17	15	12
SkinSeg	0.9482	0.9516	0.9406	0.9425
	18	13	9	8
USCI	0.9356	0.9382	0.9404	0.9396
	215	199	104	63
Cover type	0.4984	0.4987	0.4979	0.4986
	402	221	207	264
HT Sensor	0.4960	0.5008	0.4972	0.4978
	55	23	16	23
Poker Hand	0.4977	0.4993	0.4982	0.5006
	123	67	65	62
Gas Sensor	0.9865	0.9887	0.9866	0.9846
	620	287	258	254
HEPMASS	0.7929	0.7949	0.7927	0.7902
	8752	3275	3263	3321

Table 4. Table 4: Clustering accuracy and running time on UC Irvine data under 3 simulation settings, D 1 , D 2 subscript 𝐷 1 subscript 𝐷 2 D_{1},D_{2} and D 3 subscript 𝐷 3 D_{3} , when rpTrees is used as the DML. The time is in seconds. There are two distributed sites under all of D 1 , D 2 subscript 𝐷 1 subscript 𝐷 2 D_{1},D_{2} and D 3 subscript 𝐷 3 D_{3} .

Data set	Non-distributed	${𝑫}_{\mathrm{𝟏}}$	${𝑫}_{\mathrm{𝟐}}$	${𝑫}_{\mathrm{𝟑}}$
Connect-4	0.6577	0.6560	0.6554	0.6550
	7	6	5	5
SkinSeg	0.9492	0.9535	0.9434	0.9432
	5	4	3	3
USCI	0.9394	0.9391	0.9394	0.9371
	36	33	23	19
Cover type	0.4978	0.4984	0.4956	0.4968
	81	50	52	53
HT Sensor	0.4957	0.4963	0.4893	0.4836
	26	10	7	7
Poker Hand	0.4990	0.4999	0.4976	0.4993
	47	26	25	24
Gas Sensor	0.9828	0.9850	0.9811	0.9792
	305	174	175	169
HEPMASS	0.7906	0.7920	0.7902	0.7890
	1568	695	688	712

Table 5. Table 5: Simulation setting for HEPMASS data with multiple distributed sites.

# Sites	Settings	Site configuration
2	$D_1$	Site1: $C_1$,  Site2: $C_2$
	$D_2$	Site1: $0.7C_1+0.3C_2$,  Site2: $0.3C_1+0.7C_2$
	$D_3$	Data randomly partitioned among sites
3	$D_1$	Site1: $C_1/2$,  Site2: $C_1/2$,  Site3: $C_2$
	$D_2$	Site1: $C_1/2+C_2/4$,  Site2: $C_1/4+C_2/4$,
		Site3: $C_1/4+C_2/2$
	$D_3$	Data randomly partitioned among sites
4	$D_1$	Site1: $C_1/2$,  Site2: $C_1/2$,  Site3: $C_2/2$,
		Site4: $C_2/2$
	$D_2$	Site1: $3/8C_1+C_2/8$,  Site2: $3/8C1+C_2/8$,
		Site3: $C_1/8+3/8C_2$,  Site4: $C_1/8+3/8C_2$
	$D_3$	Data randomly partitioned among sites

Table 6. Table 6: Clustering accuracy and running time (in seconds) on the HEPMASS data under settings, D 1 , D 2 subscript 𝐷 1 subscript 𝐷 2 D_{1},D_{2} and D 3 subscript 𝐷 3 D_{3} , respectively. The subscripts next to the DML are the number of distributed sites.

Non-distributed	DML	${𝑫}_{\mathrm{𝟏}}$	${𝑫}_{\mathrm{𝟐}}$	${𝑫}_{\mathrm{𝟑}}$
	kmeans2	0.7949	0.7927	0.7902
		3275	3263	3321
0.7929	kmeans3	0.7937	0.7913	0.7904
8752		3159	2469	2106
	kmeans4	0.7905	0.7887	0.7891
		1474	1413	1465
	rptrees2	0.7920	0.7902	0.7890
		695	688	712
0.7906	rptrees3	0.7895	0.7900	0.7866
1568		678	564	526
	rptrees4	0.7876	0.7884	0.7869
		387	396	401

Equations

$$\mathcal{W}(V_1,V_2)=\sum _{i\in V_1,j\in V_2}{a}_{ij}\text{mbox}for{V}_1,V_2\subseteq V.$$

$$\arg \underset{V_1,...,V_K\subseteq V}{\min }\sum _{j=1}^K\frac{\mathcal{W}(V_j,V)-\mathcal{W}(V_j,V_j)}{\mathcal{W}(V_j,V)}.$$

$${\mathcal{L}}_A=D^{-\frac{1}{2}}(D-A)D^{-\frac{1}{2}},$$

$$\rho \le \Vert {\tilde{\mathbf{v}}}_2-{\mathbf{v}}_2{\Vert }^2\le ||\tilde{\mathcal{L}}-\mathcal{L}|{|}_F^2,$$

$$G=(1-\pi )\cdot G_1+\pi \cdot G_2,$$

$$\tilde{X}=X+ϵ,$$

$$\begin{gathered} ||\tilde{\mathcal{L}}-\mathcal{L}|{|}_F\le ||D^{-\frac{1}{2}}E{D}^{-\frac{1}{2}}|{|}_F \\ +(1+o(1))||\Delta D^{-\frac{3}{2}}A{D}^{-\frac{1}{2}}|{|}_F. \end{gathered}$$

$$(a-b{)}^2\le 2(a^2+b^2),(a-b{)}^4\le 8(a^4+b^4).$$

$$||\tilde{\mathcal{L}}-\mathcal{L}|{|}_F^2{\le }_{p}C\sum _{s=1}^S{\gamma }_s{\sigma }_s^2+C^{\prime }\sum _{s=1}^S{\gamma }_s{\sigma }_s^4$$

$$||D^{-\frac{1}{2}}E{D}^{-\frac{1}{2}}|{|}_F^2$$

$$||\Delta D^{-\frac{3}{2}}A{D}^{-\frac{1}{2}}|{|}_F^2$$

$$\delta (R)=\underset{q:R(q)\le R}{\inf }\mathcal{D}(q).$$

$${\delta }_d(R)\cong b_{s,d}\cdot ||f|{|}_{d/(d+s)}\cdot n^{-s/d},$$

$$||f|{|}_{d/(d+s)}={\left(\int f^{d/(d+s)}(x)dx\right)}^{(d+s)/d}.$$

$$C\cdot \underset{s=1,...,S}{\max }||f_s|{|}_{d/(d+2)}.k^{-2/d}+O(k^{-4/d}),$$

$$\underset{\tau \in \Pi }{\max }\left\{\frac{1}{N}\sum _{i=1}^N\mathbb{I}\{\tau (h(x_i))=\hat{h}(x_i)\}\right\},$$

$$\Sigma =\left[\begin{array}{cc}3& 1\\ 1& 3\end{array}\right].$$

$$\frac{1}{4}\sum _{i=1}^4\mathcal{N}({\mu }_i,\Sigma )$$

$${\mu }_1=(2.5,0,...,0),{\mu }_2=(0,2.5,0,...,0),$$

$${\mu }_3=(0,0,2.5,0,...,0),{\mu }_4=(0,0,0,2.5,0,...,0),$$

$${\Sigma }_{i,j}={\rho }^{|i-j|},\text{mbox}for\rho =0.1,0.3,\mathrm{0.6.}$$

$$\underset{S_1,S_2,...,S_K}{\mathrm{argmin}}\sum _{i=1}^K\sum _{\mathbf{x}\in S_i}{\Vert \mathbf{x}-{\mathbf{\mu }}_i\Vert }^2,$$

$$\arg\min_{j\in\{1,2,...,K\}}\big{\|}x-m_{j}\big{\|}^{2};$$

$$m_j^{new}=\frac{1}{\Vert S_j\Vert }\sum _{x\in S_j}x,j=1,2,...,K;$$

Full text

1 Introduction

Spectral clustering [45, 41, 50, 32, 56] refers to a class of clustering algorithms that work on the eigen-decomposition of the Gram matrix formed by the pairwise similarity of data points. It is widely acknowledged as the method of choice for clustering, due to its typically superior empirical performance, its flexibility in capturing a range of geometries such as nonlinearity and non-convexity [41], and its nice theoretical properties [49, 17, 26, 51, 54]. Spectral clustering has been successfully applied to a wide spectrum of applications, including parallel processing [47], image segmentation [45, 23], robotics [42, 9], web search [10], spam detection, social network mining [52, 40, 34], and market research [11] etc.

Most existing spectral clustering algorithms are “local” algorithms. That is, they assume all the data are in one place. Then either all computation are carried out on a single machine, or the data are split and delegated to a number of nodes (a.k.a. machines or sites) for parallel computation [14, 30]. It is possible that the data may be initially stored at several distributed nodes but then pushed to a central server which split and re-distribute the data. Such a case is also treated as local, as all the data are in one place at a certain stage of the data processing. However, with the emergence of big data, it is increasingly often that the data of interest are distributed. That is, the data are stored at a number of distributed sites, as a result of diverse data collection channels or business operation etc. For example, a major retail vendor, such as Walmart, has sales data collected at walmart.com, or its Walmart stores, or its warehouse chains—Sam’s Club etc. Such data from different sources are distributed as they are owned by different business groups, even all within the same corporation. Indeed there is no one central data center at Walmart, rather the data are either housed at Walmart’s Arkansas headquarter, or its e-commerce labs in the San Francisco Bay area, CA, due possibly to historical reason—Walmart was a pioneer of commercial vendors for a large scale adoption of digital technology in the late seventies to early eighties, while started its e-commerce business during the last decade. Many applications, however, would require data mining or learning of a global nature, that is, to use data from all the distributed sites, as that would be a more faithful representation of the real world or would yield better results due to a larger data size.

There are several challenges in the spectral clustering of data across distributed sites. The data at individual sites may be large. Many existing divide-and-conquer type of algorithms [14, 30] would collect data from distributed sites first, re-distribute the working load and then aggregate results computed at individual sites. The communication overhead will be high. Moreover, the data at individual sites may not have the same distribution. Additionally, the owners of the data at individual sites may not be willing to share either because the data entails value or the data itself may be too sensitive to share (not the focus of this work though). Now the question becomes, can we carry out spectral clustering for data distributed over a number of sites without transmitting large amount data?

One possible solution is to carry out spectral clustering at individual sites and then ensemble. However, as the data distribution at individual sites may be very different, and to ensemble needs distributional information from individual sites which is often not easy to derive. Thus, an ensemble type of algorithm will not work, or, at least not in a straightforward way. Another possibility is to modify existing distributed algorithms, such as [14, 30], but that would require support from the computing infrastructure—to enable a close coordination and frequent communication of intermediate results among individual nodes—which is not easy to implement and the solution may not be generally applicable.

Our proposed framework is based on the distortion minimizing local (DML) transformations [56]. The idea is to generate a small set of codewords (a.k.a. representative points) that preserve the underlying structure of the data at individual nodes. Such codewords are then transmitted to a central server (or any one of the nodes) where spectral clustering is carried out, and the result from spectral clustering will be populated to individual nodes so that the cluster membership of all points can be recovered according to correspondence information maintained at individual nodes. As the codewords from individual sites preserve the geometry of the data, the expected loss in clustering accuracy w.r.t. that where all the data are in one place would be small. The key benefit of our approach lies in the fact that the generation of codewords is local in the sense that no distributional information from other sites is required. Additionally, computation of the codewords can be carried out in parallel on individual nodes thus speeds up the overall computation and meanwhile makes the best use of the existing computing resources.

Our contributions are as follows. Motivated by emerging applications in distributed computing, we propose a new line of research—spectral clustering when the data are distributed over multiple sites. The DML-based framework we propose readily facilitates data mining tasks such as spectral clustering over distributed data while eliminating the need of big data transmission. Our approach is theoretically sound; the loss in the accuracy of spectral clustering, when compared to that in a non-distributed setting, vanishes when increasing the size of data transmission. Major computations are naturally localized and parallelized in the sense that they are carried out on the node where the data is stored, which would lead to a major speedup in the overall computation. Additionally, as each node is working on part of the full data, this has the potential benefit of divide-and-conquer in the sense that even if we add up the computation time at all individual nodes, the total time may still be less than that in a non-distributed setting for big enough data. While this work deals with spectral clustering for concreteness and for being an important methodology at its own right, really it represents a new paradigm in distributed computing, that is, data mining or inference over distributed data with major computation carried out at where the data are located while achieving the effect of using the full data at a “minimal” communication cost.

The remaining of this paper is organized as follows. In Section 2, we will describe our framework and explain its implementation. This is followed by a discussion of related work in Section 3. In Section 4, we present theoretical analysis of our proposed approach. In Section 5, we show experimental results. Finally we conclude in Section 6.

2 A framework for spectral clustering on distributed data

The idea underlying our approach is the notion of continuity. That is, similar data would play a similar role in learning and inference, including clustering. Such a notion suggests a class of data transformations called distortion minimizing local (DML) transformation. The idea of DML transformation is to represent the data by a small set of representative points (or codewords). One can think of this as a “small-loss” data compression, or the representative set as a sketch of the full data. Since the representative set resembles the full data, learning based on it is expected to be close to that on the full data. Similar idea was explored in [56, 5] and has been successfully applied to some computation-intensive algorithms, assuming the data are non-distribted. In all these previous work, DMLs were mainly introduced to address the computational challenge.

DMLs can be used to enable spectral clustering over distributed data, that is, the data are not stored in one machine but over a number of distributed nodes. Our framework for spectral clustering over distributed data is surprisingly simple to implement. It consists of three steps:

Apply DML to data at each distributed node

• 2)

Collect codewords from all nodes, and carry out spectral clustering on the set of all codewords

• 3)

Populate the learned clustering membership by spectral clustering back to each distributed node.

Figure 1 is an illustration of our framework. It is clear that algorithms designed under this framework would eliminate the need of having to transmit large amount of data among distributed nodes—only those codewords need to be transmitted. As the codewords are DML-transformed data, data privacy may be ensured since no original data are transmitted. Additionally, as spectral clustering is only performed on the set of codewords, the overall computation involved will be greatly reduced. Also as the DMLs and the recovering of cluster membership are performed at individual nodes, such computation can be done in parallel. We start by a brief introduction to spectral clustering.

2.1 Introduction to spectral clustering

Spectral clustering works on an affinity graph over data points $X_{1},...,X_{N}$ and seeks to find a “minimal” graph cut. Depending on the choice of the similarity metric and the objective function to optimize, there are a number of variants, including [45, 41]. Our discussion will be based on normalized cuts [45].

An affinity graph is defined as a weighted graph $\mathcal{G}=(V,\mathcal{E},A)$ where $V=\{X_{1},...,X_{N}\}$ is the set of vertices, $\mathcal{E}$ is the edge set, and $A=(a_{ij})_{i,j=1}^{N}$ is the affinity matrix with $a_{ij}$ encoding the similarity between $X_{i}$ and $X_{j}$. Figure 2 is an illustration of graph cut.

Let $V=(V_{1},\ldots,V_{K})$ be a partition of $V$. Define the size of the cut between $V_{1}$ and $V_{2}$ by

[TABLE]

Normalized cuts seeks to find the minimal (normalized) graph cut, or equivalently, solve an optimization problem

[TABLE]

The above is an integer programming problem thus intractable, a relaxation to real values leads to an eigenvalue problem for the Laplacian matrix111In the following, we will omit the subscript $A$ when no confusion is caused.

[TABLE]

where $D=diag(d_{1},...,d_{N})$ is the degree matrix with $d_{i}=\sum_{j=1}^{N}a_{ij},i=1,...,N$. Normalized cuts look for the second smallest eigenvector of $\mathcal{L}_{A}$, and round its components to produce a bipartition of the graph. Similar procedure is applied to each of the bipartitions recursively until reaching the number of predefined clusters.

2.2 Distortion minimizing local transformation

A key property that makes DMLs applicable to distributed data is being local. That is, such a data transformation can be done locally, without having to see the full data. Thus DML can be applied at individual distributed nodes separately. If one can pool together all those codewords, then overall inference or data mining can be easily carried out. Thus, as long as the local data transformations are fine enough, a large class of inference or data mining tools will be able to yield result as good as using the full data.

As we are dealing with big volumes of data, a natural requirement for a DML is its computational efficiency while incurring very “little” loss in information. We will briefly describe two concrete implementations of the DML transformation, one by $K$-means clustering and the other by random projection trees (rpTrees), both proposed in [56].

2.2.1 K-means clustering

$K$-means clustering was developed by S. Lloyd in 1957 [38], and remains one of the simplest yet popular clustering algorithms. The goal of $K$-means clustering is to split data into $K$ partitions (clusters) and assign each point to the “nearest” cluster (mean). A cluster mean is the center of mass of all points in a cluster; it is also called cluster centroid or codewords. The algorithm starts with a set of randomly selected cluster centers, then alternates between two steps: 1) assign all points to their nearest cluster centers; 2) recalculate the new cluster centers. The algorithm stops when the change is small according to a cluster quality measure. For a more details about K-means clustering, please refer to the appendix and [29, 38].

When DML is implemented by $K$-means clustering, each of the distributed nodes performs $K$-means clustering separately. Note that, the number of clusters, $K$, may be different for a different node; the only requirement is that $K$ is reasonably large. The set of representative points are taken as the cluster centers, or average of points in the same cluter by an appropriate metric. Empirically, the computation of $K$-means clustering scales linearly with the number of data points, thus it is suitable for big data.

2.2.2 Random projection trees

To see how the K-D tree [8] can be used for a DML transformation, we first describe how a K-D tree grows. Let the collection of all data at a distributed site correspond to the root node, $\mathcal{D}^{(0)}$, of a tree. Now one variable, say $v$ (index of a variable), is selected to split the root node; the root node is split into two child nodes, $\mathcal{D}_{L}^{(1)}$ and $\mathcal{D}_{R}^{(1)}$, according to whether a data points has its $v$-th coordinate smaller or larger than a cutoff point. For each of $\mathcal{D}_{L}^{(1)}$ and $\mathcal{D}_{R}^{(1)}$, we will follow a similar procedure recursively. This process continues until some stopping criterion is met. Figure 3 illustrates the construction of a K-D tree. As the tree construction can be viewed as generating a recursive space partition [8, 19, 55] of the data, data points falling in the same leaf node would be similar (if the leaf node is “small”) and their average can be used as the representative. If the underlying data has a very high dimension, then a randomized variant, rpTrees [19], can be used which would adapt to the geometry of the underlying data and effectively overcome the curse of dimensionality. In this work, we use rpTrees implemented in [59].

2.3 Algorithmic description

Now we can briefly describe how to adopt spectral clustering for distributed data. Assume that there are $S$ distributed sites. Apply DML (K-means clustering or rpTrees) at each site individually. Let $Y_{i}^{(s)},i=1,2,...,n_{s}$ be the group centroids of data at site $s=1,2,...,S$. A group is either data in the same cluster if K-means clustering is used, or all points in the same leaf node of rpTrees. The centroids are center of mass of all points in the same group. Spectral clustering is performed on the set of group centroids (representative points) collected from all the $S$ sites. An algorithmic description is given in Algorithm 1.

If the DML transformation is linear, which is the case if it is implemented by K-means clustering or rpTrees, then the overall computational complexity is easily seen to be linear in the total number of points in the distributed data. Indeed for large scale distributed computation, that is an implicit requirement.

3 Related work

There has been an explosive growth of interests in distributed computing in the last decades. One driving force behind is the prevalence of low-cost clustered computers and storage systems [2, 24], which makes it feasible to interconnect hundreds or thousands of clustered computers. Numerous systems and computing platforms have been developed for distributed computing. For example, Google’s Bigtable [25, 12], the Apache Hadoop/Map-Reduce [20, 46], the Spark system [62, 61], and Amzon’s AWS cloud etc. The literature is huge, but mostly on distributed system architecture, computing platforms, or data query tools. For a review of recent developments, please see [13, 21] and references therein.

Existing distributed algorithms in the literature are either parallel algorithms, for example, [14], or use a divide-and-conquer strategy which split the data and distribute the workload to a number of nodes [30]. An influential line of work is Bag of Little Bootstrap [33]. This work aims at computing a big data version of Bootstrap [22], a fundamental tool in statistical inference; the idea is to take many very “thin” subsamples, distribute the computing on each subsample to a node and then aggregate results from those individual subsamples. Recently, [7] considered the general distributed estimation and inference in the Divide-and-Conquer paradigm, and obtained optimal partitions for data. Chen and Xie [16] studied penalized regression and model selection consistency when the data is too big to fit into the memory of a single machine by working on a subsample of the data and then aggregating the resulting models. Singh et al [48] designed DiP-SVM, a distribution preserving kernel support vector machine where the first and second order statistics of the data are retained in each of the data partitions, and run spectral clustering on each data partition at an individual nodes with results aggregated. One thing common about these work is that all assume one has the entire data before delegating the workload to individual nodes or pushing the data to temporary storage and then aggregate the individual results; the data are distributed or split mainly for improving computational efficiency or solving the memory shortage problem.

Many works have been proposed to improve the efficiency of spectral clustering. Chen and Cai [15] proposed a landmark-based method for spectral clustering by selecting representative data points as a linear combination of the original data. Zhang and his co-authors [63] proposed an incremental sampling approach, i.e., the landmark points are selected one at a time adaptively based on the existing landmark points. Liu et al. [37] proposed a fast constrained spectral clustering algorithm via landmark-based graph construction and then reduce the data size by random sampling after spectral embedding. Paiva [43] proposed to select a representative subset of the training sample with an information-theoretic strategy. Lin et al [36] designed a scalable co-association cluster ensemble framework using a compressed version of co-association matrix formed by selecting representative points of the original data.

Several recent work aims at reducing the data volume by deep learning. Aledhari et al [1] proposed a deep learning based method to minimize large genomic DNA dataset for transmission through the internet. Banijamali et al [6] integrated the recent deep auto-encoder technique into landmark-based spectral clustering.

4 Analysis of the algorithm

In our distributed framework, each node $s$ individually applies DML to generate codewords $\mathbb{Y}_{s}=\{Y_{i}^{(s)}:i=1,2,...,n_{s}\}$, to be sent to a central node for spectral clustering. The results from spectral clustering are returned to node $s$ for the recovery of cluster membership for all points at node $s=1,...,S$. A crucial question is, does this approach work? Since each site performs DML locally and no site is using distributional information from others. How much additional error will be incurred under our framework, or will such an error vanish if the data is big? The goal of our analysis is to shed lights to these questions.

To carry out such an analysis is challenging. We are interested in the clustering error (an end-to-end error), but we only observe errors in terms of the local distortion in representing the points, $\mathbb{X}_{s}=\{X_{i}^{(s)}:i=1,2,...,N_{s}\}$ by codeword $\mathbb{Y}_{s}$ for each $s=1,...,S$. We need to establish a connection between the clustering error and the distortion in data representation of a distributed (local) nature.

A second challenge is related to the local ‘optimality’ of our algorithm. That is, each set of codewords $\mathbb{Y}_{s}$ is an ‘optimal’ representation of data, $\mathbb{X}_{s}$, at a local site, but their union, $\mathbb{Y}=\cup_{s=1}^{S}\mathbb{Y}_{s}$, is not necessary the optimal set of codewords for the entire set of data $\mathbb{X}=\cup_{s=1}^{S}\mathbb{X}_{s}$. One situation for which this happens is when there is an overlap among the support of points from different sites. As illustrated in Figure 4, the original codes $A_{1},A_{2}$ are no longer the optimal codewords for the combined data; rather the new optimal codewords become $B_{1}$ and $B_{2}$. Now the question is, will local ‘optimality’ at all individual sites be sufficient or if additionally constraints may be required to make our approach work?

One crucial insight to our analysis is, what we really need is the convergence of the global distortion to zero when the size of data increases, not necessary the optimality of the global distortion. So we would proceed as if all the data are from one node, and only plugs in analysis of local DMLs when necessary. Our analysis hinges on an important result obtained in [56] which establishes the connection between the end-to-end error and the perturbations to the Laplacian matrix due to data distortion.

Lemma 1** ([56]).**

Under assumptions $\mathbb{A}_{1-3}$, the mis-clustering rate $\rho$ of a spectral bi-partitioning algorithm on perturbed data satisfies

[TABLE]

where $||.||_{F}$ indicates the Frobenius norm [27].

By Lemma 1, in order to bound the additional clustering error resulting from the distributed nature of the data, we just need to bound the distortion of the Laplacian matrix due to a compressed data representation by codewords from a number of distributed sites.

We will proceed in two steps. First, we conduct a perturbation analysis of the Laplacian matrix. Then, we will adopt results from local DMLs to the perturbation results. Note that in the perturbation analysis, we treat all the data as if they are all from one site. We follow notations in [56].

Our theoretical model is the following two-component Gaussian mixture

[TABLE]

where $\pi\in\{0,1\}$ with $\mathbb{P}(\pi=1)=\eta$. The choice of a two-component Gaussian mixture is mainly for simplicity; it should be clear that our analysis applies to any finite Gaussian mixtures. We treat data perturbation as adding a noise component $\epsilon$ to $X$:

[TABLE]

and we denote the distribution of $\tilde{X}$ by $\tilde{G}$. We assume $\epsilon$ is symmetric about [math] with bounded support, and let $\epsilon$ has a standard deviation $\sigma_{\epsilon}$ that is small compared to $\sigma$, the standard deviation for the distribution of $X$.

4.1 Perturbation analysis

Our perturbation analysis relies on one more result established in [56] which is stated as Lemma 2.

Lemma 2** ([56]).**

Let $\mathcal{L}$ and $\tilde{\mathcal{L}}$ be the Laplacian matrix corresponding to the original similarity matrix and that after perturbation. Then

[TABLE]

We state without proof some elementary results.

Lemma 3**.**

Let $a,b\in\mathbb{R}$. Then the following holds

[TABLE]

The main result of our perturbation analysis is stated as Theorem 1.

Theorem 1**.**

Suppose $X_{1},...,X_{N}\in\mathbb{R}^{d}$ are generated i.i.d. according to (2) such that $\inf_{1\leq i\leq N}d_{i}/N>\delta_{0}$ holds in probability for some constant $\delta_{0}>0$. Further, the number of data points, $N_{s}$, at site $s$ carries a substantial fraction of the total number of data points $N$ in the sense that $\lim_{N\rightarrow\infty}N_{s}/N=\gamma_{s}\in(0,1)$ for $s=1,...,S$. Assume the data perturbation $\epsilon$ is symmetric about [math] with bounded support. Further assume $||\Delta D^{-1}||_{2}=o(1)$. Then

[TABLE]

for some constants $C$ and $C^{\prime}$, as $N\rightarrow\infty$.

Proof.

Our proof essentially follows that of Theorem 5 and Theorem 6 in [56] except during the final steps of Lemma 13 and Lemma 14. Instead of using theory of U-statistics [31] which would give a sharper bound, we use an upper bound, i.e., Lemma 3, which allows us to group the perturbation error of individual data points by the distributed site they belong to. By adopting the proof of Lemma 13, we have

[TABLE]

as $N\rightarrow\infty$. In the above $\epsilon_{i}$ is the perturbation to observation $X_{i}$ for $i=1,...,N$. Similarly in the proof of Lemma 14, we have

[TABLE]

as $N\rightarrow\infty$. Combining the above two inequalities and then apply Lemma 2, we have proved the theorem. ∎

4.2 Quantization errors by K-means clustering

Existing work from vector quantization [60, 28] allows us to characterize the amount of distortion when the representative set is computed by $K$-means clustering.

Let a quantizer $q$ be defined as $q:\mathbb{R}^{d}\mapsto\{y_{1},\ldots,y_{n}\}$ for $y_{i}\in\mathbb{R}^{d}$. For $X$ generated from a random source in $\mathbb{R}^{d}$, let the distortion of $q$ be defined as: $\mathcal{D}(q)=\mathbb{E}||X-q(X)||^{s}$, which is the mean square error for $s=2$. Let $R(q)=\log_{2}n$ denote the rate of the quantization code. Define the distortion-rate function $\delta(R)$ as

[TABLE]

Then $\delta(R)$ can be characterized in terms of the source density $f(\cdot)$, and constants $d,s$ by the following theorem.

Theorem 2** ([60, 28]).**

Let $f(x)$ be the $d$-dimensional probability density. Then, for large rates $R$, the distortion-rate function of fixed-rate quantization can be characterized as:

[TABLE]

where $\cong$ means the ratio of the two quantities tends to 1, $b_{s,d}$ is a constant depending on $s$ and $d$, and

[TABLE]

Now we can apply Theorem 2 and Pollard’s strong consistency of K-means clustering [44] to individual terms in Theorem 1, and establish a performance bound for our approach.

Theorem 3**.**

Assume the data distributions in a S-site distributed environment have density $f_{1},...,f_{S}$, respectively. Let $k_{1},...,k_{S}$ be the number of codewords at the $S$ distributed sites. Let $k=min(k_{1},...,k_{S})$. Then the additional clustering error rate $\rho$, as compared to non-distributed setting, under our framework can be bounded by

[TABLE]

where $C$ is a constant determined by the number of clusters, the variance of the original data, the bandwidth of the Gaussian kernel and the eigengap of Laplacian matrix (or minimal eigengap of the Laplacian of all affinity matrices used in normalized cuts).

5 Experiments

In this section, we report our experimental results. This includes simulation results on synthetic data, and on data from the UC Irvine Machine Learning Repository [35]. We will compare the performance of distributed vs non-distributed (where all the data are assumed to be in one place). The spectral clustering algorithm used is normalized cuts [45], and the Gaussian kernel is used in computing the affinity (or Gram) matrix with the bandwidth chosen via a cross-validatory search in the range (0, 200] (with step size 0.01 within (0,1], and 0.1 over (1,200]) for each data set. All algorithms are implemented in R programming language, and the kmeans() function in R is used for which details are similar as those documented in [56].

The metric for clustering performance is clustering accuracy, which counts the fraction of labels given by a clustering algorithm that agree with the true labels (or labels come with the dataset). Let $\{1,...,K\}$ denote the set of class labels, and $h(.)$ and $\hat{h}(.)$ are the true label and the label given by the clustering algorithm, respectively. The clustering accuracy is defined as [56]

[TABLE]

where $\mathbb{I}$ is the indicator function and $\Pi$ is the set of all permutations on the class labels $\{1,...,K\}$. While there are dozens of performance metrics around for clustering, the clustering accuracy is a good choice for the evaluation of clustering algorithms. This is because the label (or cluster membership) of individual data points is the ultimate goal of clustering, while other clustering metrics are often a surrogate of the cluster membership, and they are used in practice mainly due to the lack of labels. For the evaluation of clustering algorithms, we do have the freedom of using those datasets coming with a label. Indeed the clustering accuracy is commonly used for the evaluation of clustering; see, for example, [53, 4, 56].

We also consider the time for computation. The elapsed time is used. It is counted from the moment when the data are loaded into the memory (R running time environment), till the time when we have obtained the cluster label for all data points. We assume all the distributed nodes run independently, so the longest computation time among all the sites is used (instead of adding them up). As we do not have multiple computers for the experiments, we do not explore the communication time for transmitting representative points and the clustering results. Indeed, for the dataset used in our experiments, such time could be ignored when compared to the computation time, as the number of representative points are all less than 2000. The time reported in this work are produced on a MacBook Air laptop computer, with 1.7GHz Intel Core i7 processor and 8G memory.

5.1 Synthetic data

To appreciate how those representative points look like, we first consider a toy example.

The data is generated by a 4-component Gaussian mixture $\frac{1}{4}\sum_{i=1}^{4}\mathcal{N}(\mu_{i},\Sigma)$ with $\mu_{1}=(2,2)$, $\mu_{2}=(-2,-2)$, $\mu_{3}=(-2,2)$ and $\mu_{4}=(2,-2)$, and the covariance matrix given by

[TABLE]

The Gaussian mixture is commonly used as a generating model for clusters in the literature [18, 39, 51, 58, 54], due to its simplicity (as the mixture components are Gaussians) and also because it is amenable for theoretical analysis. Indeed, it is used as the theoretical model in our analysis (c.f. Section 4). Note that here we are not arguing that spectral clustering is superior to other clustering methods, rather our claim was spectral clustering could work in a distributed setting that we consider.

Figure 5 is a scatter plot of the Gaussian mixture. The four mixture components are marked by blue, red, green, and cyan, respectively. Here in generating representative points, we run $K$-means clustering on the two sites defined by $\mathcal{N}(\mu_{1},\Sigma)+\mathcal{N}(\mu_{2},\Sigma)$ and $\mathcal{N}(\mu_{3},\Sigma)+\mathcal{N}(\mu_{4},\Sigma)$, respectively. It can be seen that those representative points serve as a “good” condensed representation of the original data points.

We conduct experiments on a 4-component Gaussian mixture on $\mathbb{R}^{10}$

[TABLE]

with the center of the four components being

[TABLE]

and the covariance matrix $\Sigma$ defined by

[TABLE]

For simplicity, we only consider the case with two sites in a distributed environment. It should be easy to extend to the general case. Let $\mathcal{C}_{i},i=1,2,3,4$, denote the data from the four mixture components, respectively. To simulate how the data may look like in a distributed environment, we create the following three scenarios:

• $D_{1}$:

Site 1 has $\mathcal{C}_{1}+\mathcal{C}_{2}$, Site 2 has those from $\mathcal{C}_{3}+\mathcal{C}_{4}$

• $D_{2}$:

Site 1 has $\frac{1}{2}\mathcal{C}_{1}+\mathcal{C}_{2}+\frac{1}{2}\mathcal{C}_{3}$, Site 2 has $\frac{1}{2}\mathcal{C}_{1}+\frac{1}{2}\mathcal{C}_{3}+\mathcal{C}_{4}$

• $D_{3}$:

Site 1 and 2 each has a randomly selected half of the data points

where $\frac{1}{2}\mathcal{C}_{1}$ means that a distributed site contains half of the data points from component $\mathcal{C}_{1}$, and so on. Note these scenarios are not the different ways that we split and distribute the data for fast computation rather each should be viewed as one type of distributed settings: $D_{1}$ for which data at different sites have roughly disjoint supports, $D_{2}$ for which data at different sites have some overlap in terms of supports, and in $D_{3}$ individual sites have similar data distribution. 40000 data points are generated from the Gaussian mixture (6), and the number of representative points is $1000$ (i.e., the data compression ratio is 40:1). The number of data points at Site 1 and 2, and also the number of representative points can all be calculated by the site specification and the data compression ratio accordingly. Note that here the number of clusters in K-means clustering (or the number of leaf nodes in rpTrees) is no longer as important as that in usual clustering, as long as the resulting groups are sufficiently fine (i.e., data points in the same group are “similar”); since here K-means clustering (or rpTrees) is used to group the data for efficient distributed computation. The same discussion applies to UC Irvine datasets as well.

Figure 6 and Figure 7 show the clustering accuracy of spectral clustering under various distributed scenarios as compared to non-distributed when K-means clustering and rpTrees are used as DML, respectively. Here the true label of each data point is taken as its mixture component ID. The data compression ratio for K-means clustering is set to be 40:1, which is the ratio of the size of the data to the number of clusters in DML; the maximum size of the leaf nodes is 40 for rpTrees, to match approximately the data compression ratio in K-means clustering. In all cases, the accuracy computed for distributed data is close to non-distributed. The reason that the accuracy under $D_{1}$ may be higher than others (including the non-distributed) may be attributed to the fact the data are less mixed than in other settings thus implicitly achieving a regularization effect where data are grouped within the same classes or the part of data that are less mixed than others (one can roughly view this as additional constraints on clustering).

5.2 UC Irvine data

The UC Irvine datasets we use include the Connect-4, SkinSeg (Skin segmentation), USCI (US Census Income), Cover type, HT Sensor, Poker Hand, Gas Sensor and the HEPMASS data. Table 1 gives a summary of the datasets. For Connect-4, USCI, and Poker Hand data, we follow procedures described in [56] to preprocess the data. The original USCI data has 299,285 instances with 41 features. We excluded instances with missing values, and also features #26, #27, #28 and #30, due to too many values. This leaves 285,799 instances on $37$ features, with all categorical variables converted to integers. The original Cover Type data has 581,012 instances. We excluded the two small classes (i.e., 4 and 5) for fast evaluation of accuracy (otherwise all 7! permutations need to be evaluated, but that is not our focus of the feasibility of distributed computing), and this leaves 568,772 instances; we also standardized the first 10 features such that each has a mean 0 and variance 1. The original Poker Hand data is highly unbalanced, with $6$ small classes containing less than $1\%$ of the data. Merging small classes gives $3$ final classes with a class distribution of $50.12\%$, $42.25\%$ and $7.63\%$, respectively. The Gas Sensor data consists of two different gas mixtures: Ethylene mixed with CO, and Ethylene mixed with Methane. The data corresponding to these two gas mixtures form the two classes in the data. The Connect-4, the USCI, and the Gas Sensor data are normalized such that all features have mean [math] and standard deviation $1$.

It is worthwhile to remark that, though the datasets we use may be smaller than some large data at industry-scale, the Gas Sensor and the HEPMASS are among the largest datasets currently available in the UC Irvine data repository. Moreover, in industry often the raw data used may be huge but, after preprocessing and feature engineering, the data used for data mining or model building are substantially smaller. Additionally, the UC Irvine datasets used in our experiments have a wide variety of sizes, ranging from 67,557 to 10,500,000. We believe those are sufficient to demonstrate the feasibility of our algorithm for distributed computing; for even larger data, results from our theoretical analysis (large sample asymptotics) can be used to gain insights.

Similar as the synthetic data, we conduct experiments under three distributed settings for each UC Irvine dataset. Table 2 lists the site specification under such settings. These settings are all common in practice. $D_{1}$ corresponds to situations where different sites have data with disjoint supports, and $D_{2}$ the case where data from different sites are mixed in terms of the support of the data, while $D_{3}$ is for the case where the data at each site is a random sample from a common data distribution. Each of $D_{1}$, $D_{2}$ and $D_{3}$ in Table 2 corresponds to two columns with the left indicating the data composition for Site 1 and the right for Site 2 (under $D_{3}$ each of the two distributed sites has a random half of the full data). The number of data points at Site 1 and 2, and also the number of representative points can all be calculated by the site specification and the data compression ratio accordingly.

Table 3 show the clustering accuracy and elapsed time for each UC Irvine dataset under settings $D_{1},D_{2}$, and $D_{3}$, respectively, when K-means clustering is used as DML. The data compression ratios are 200, 800, 500, 500, 3000, 3000, 16000 and 7000, respectively. Table 4 is for rpTrees. Due to the random nature of rpTrees, it is hard to set the exact data compression ratio. To match the compression ratio for K-means clustering, we set the maximum size of leaf nodes to be 200, 800, 500, 500, 3000, 3000, 16000 and 7000, respectively. It can be seen that, for all cases, the loss in clustering accuracy due to distributed computing is small. By local computation at individual distributed sites in parallel, the overall time required for spectral clustering is significantly reduced. When the computation is evenly distributed across different sites, the saving in time is often most substantial; this is typically the case under $D_{3}$ where data are evenly split among two sites. At the similar level of data compression, DML by rpTrees is more efficient in computation than that by K-means clustering, but incurs slightly more loss in accuracy. Note that for simplicity, we consider two distributed sites for all the UC Irvine data except the HEPMASS data. For the rest of this section, we explore 3 or 4 distributed sites for the HEPMASS data.

5.2.1 Multiple sites

To study the impact of the number of distributed sites to our algorithm, we run our algorithm on the HEPMASS data for 3 or 4 distributed sites. The site configuration is shown in Table 5 where, for completeness, we also include the case of two sites from the previous section. The results are shown in Table 6. We can see that in all cases, the clustering accuracy does not or degrade very little when increasing of the number of distributed sites. For both cases, i.e., with K-means or rpTrees as the DML, the running time decreases when more sites are involved. However, the decreasing trend gradually slows down; such a phenomenon is more pronounced for the case of rpTrees as the local computation takes much less time than by K-means. This is because, as more distributed sites are involved, eventually most of the computation time would be on spectral clustering part.

6 Conclusion

We have proposed a novel framework that enables spectral clustering for distributed data, with “minimal” communication overhead while a major speedup in computation. Our approach is statistically sound in the sense that the achieved accuracy is as good as that when all the data are in one place. Our approach achieves computational speedup by deeply compressing the data with DMLs (which also sharply reduce the amount of data transmission) and leveraging existing computing resources for local parallel computing at individual nodes. The speedup in computation, compared to that in a non-distributed setting, is expected to scale linearly (with a potential of even faster when the data is large enough) with the number of distributed nodes when the data are evenly distributed across individual sites. Indeed, on all large UC Irvine datasets used in our experiments, our approach achieves a speedup of about 2x when there are two distributed sites. Two concrete implementations of DMLs are explored, including that by K-means clustering and by rpTrees. Both can be computed efficiently, i.e., computation (nearly) linear in the number of data points. One additional feature of our framework is that, as the transmitted data are not in their original form, data privacy may also be preserved.

Our proposed framework is promising as a general tool for data mining over distributed data. Methods developed under our framework will allow practitioners to use potentially much larger data than previously possible, or to attack problems previously not feasible, due to the lack of data for various reasons, such as the challenges in big data transmission and privacy concerns in data sharing.

Appendix

6.1 The K-means clustering algorithm

Formally, given $n$ data points, $K$-means clustering seeks to find a partition of $K$ sets $S_{1},S_{2},...,S_{K}$ such that the within-cluster sum of squares, $SS_{W}$, is minimized

[TABLE]

where $\mu_{i}$ is the centroid of $S_{i},i=1,2,...,K$.

Directly solving the problem formulated as (7) is hard, as it is an integer programming problem. Indeed it is a NP-hard problem [3]. The K-means clustering algorithm is often referred to a popular implementation sketched as Algorithm 2 below. For more details, one can refer to [29, 38].

6.2 The random projection tree algorithm

in this section, we given an algorithmic description of the generation of rpTrees which we adopt from [59]. Let $U$ denote the given data set. Let $t$ denote the rpTree to be built from $U$. Let $\mathcal{W}$ denote the set of working nodes. Let $n_{T}$ denote a predefined constant for the minimal number of data points in a tree node for which we will split further. Let $P_{\stackrel{{\scriptstyle\rightharpoonup}}{{r}}}(x)$ denote the projection coefficient of point $x$ onto line $\stackrel{{\scriptstyle\rightharpoonup}}{{r}}$. Let $\mathcal{N}$ denote the set of neighborhoods, and each element of $\mathcal{N}$ is a subset of neighboring points in $U$.