The Effectiveness of Uniform Sampling for Center-Based Clustering with Outliers

TL;DR

This paper introduces a new unified framework analyzing uniform sampling for center-based clustering with outliers, demonstrating its efficiency and effectiveness in reducing complexity while maintaining quality.

Contribution

It presents the first uniform sampling approach that discards exactly z outliers and provides theoretical guarantees based on a significance criterion.

Findings

Sample size can be independent of n/z ratio and dimensionality.

Uniform sampling achieves comparable results to existing methods.

Method significantly reduces running time in experiments.

Abstract

Clustering has many important applications in computer science, but real-world datasets often contain outliers. Moreover, the presence of outliers can make the clustering problems to be much more challenging. To reduce the complexities, various sampling methods have been proposed in past years. Namely, we take a small sample (uniformly or non-uniformly) from input and run an existing approximation algorithm on the sample. Comparing with existing non-uniform sampling methods, the uniform sampling approach has several significant benefits. For example, it only needs to read the data in one-pass and is very easy to implement in practice. Thus, the effectiveness of uniform sampling for clustering with outliers is a natural and fundamental problem deserving to study in both theory and practice. In this paper, we propose a new and unified framework for analyzing the effectiveness of uniform sampling for three representative center-based clustering with outliers problems, $k$-center/median/means clustering with outliers. We introduce a "significance" criterion and prove that the performance of uniform sampling depends on the significance degree of the given instance. In particular, we show that the sample size can be independent of the ratio $n/z$ and the dimensionality. More importantly, to the best of our knowledge, our method is the first uniform sampling approach that allows to discard exactly $z$ outliers for these three center-based clustering with outliers problems. The results proposed in this paper also can be viewed as an extension of the previous sub-linear time algorithms for the ordinary clustering problems (without outliers). The experiments suggest that the uniform sampling method can achieve comparable clustering results with other existing methods, but greatly reduce the running times.

Tables

Table 1. Table 1: Precision on synthetic datasets

${ϵ}_1/{ϵ}_2=$	0.6	0.8	1	2	4	6
Algorithm 1 $\tau =1$	0.512	0.734	0.755	0.746	0.774	0.876
Algorithm 1 $\tau =\frac{4}{3}$	0.808	0.897	0.839	0.999	0.999	0.852
Algorithm 1 $\tau =\frac{5}{3}$	0.874	0.845	0.973	0.999	0.999	1.000
Algorithm 1 $\tau =2$	0.974	0.997	0.997	0.998	0.999	0.999
Algorithm 2	None	None	None	1.000	1.000	1.000
MK	1.000	1.000	1.000	1.000	1.000	1.000
Malkomes	1.000	1.000	1.000	1.000	1.000	1.000
Charikar	1.000	1.000	1.000	1.000	1.000	1.000
DYW	0.999	0.997	0.997	0.998	0.998	0.997
Algorithm 3 $\tau =1$	1.000	1.000	1.000	1.000	1.000	1.000
Algorithm 3 $\tau =\frac{4}{3}$	1.000	1.000	1.000	1.000	1.000	0.999
Algorithm 3 $\tau =\frac{5}{3}$	1.000	1.000	1.000	1.000	0.999	0.999
Algorithm 3 $\tau =2$	1.000	1.000	0.999	0.999	0.997	0.998
Algorithm 4	None	None	None	1.000	1.000	1.000
$k$-means$--$	1.000	1.000	1.000	1.000	1.000	1.000
LocalSearch	1.000	1.000	1.000	1.000	1.000	1.000
DataSummary	1.000	0.999	1.000	1.000	1.000	1.000
Algorithm 3 ($k$-median) $\tau =1$	0.994	0.998	0.997	0.998	0.999	0.999
Algorithm 3 ($k$-median) $\tau =\frac{4}{3}$	1.000	1.000	0.998	1.000	1.000	1.000
Algorithm 3 ($k$-median) $\tau =\frac{5}{3}$	1.000	1.000	1.000	1.000	1.000	1.000
Algorithm 3 ($k$-median) $\tau =2$	1.000	1.000	1.000	1.000	1.000	1.000
Algorithm 4 ($k$-median)	None	None	None	1.000	1.000	1.000
$k$-means$--$ ($k$-median)	1.000	1.000	1.000	1.000	1.000	1.000
LocalSearch ($k$-median)	1.000	1.000	1.000	1.000	1.000	1.000
DataSummary ($k$-median)	0.958	0.932	0.948	0.927	0.964	0.938

Table 2. Table 2: Purity on synthetic datasets

${ϵ}_1/{ϵ}_2=$	0.6	0.8	1	2	4	6
Algorithm 1 $\tau =1$	0.746	0.804	0.722	0.977	0.896	0.978
Algorithm 1 $\tau =\frac{4}{3}$	0.860	0.981	0.931	1.000	0.999	0.976
Algorithm 1 $\tau =\frac{5}{3}$	0.957	0.953	0.893	1.000	1.000	1.000
Algorithm 1 $\tau =2$	0.972	1.000	1.000	1.000	1.000	1.000
Algorithm 2	None	None	None	1.000	1.000	1.000
MK	1.000	1.000	1.000	1.000	1.000	1.000
Malkomes	1.000	1.000	1.000	1.000	1.000	1.000
Charikar	1.000	1.000	1.000	1.000	1.000	1.000
DYW	1.000	1.000	1.000	1.000	1.000	1.000
Algorithm 3 $\tau =1$	0.972	0.989	0.986	0.986	0.961	0.914
Algorithm 3 $\tau =\frac{4}{3}$	1.000	1.000	0.998	1.000	1.000	1.000
Algorithm 3 $\tau =\frac{5}{3}$	1.000	1.000	1.000	1.000	1.000	1.000
Algorithm 3 $\tau =2$	1.000	1.000	1.000	1.000	1.000	1.000
Algorithm 4	None	None	None	1.000	1.000	1.000
$k$-means$--$	1.000	1.000	1.000	1.000	1.000	1.000
LocalSearch	1.000	1.000	1.000	1.000	1.000	1.000
DataSummary	0.958	0.932	0.948	0.927	0.964	0.938
Algorithm 3 ($k$-median) $\tau =1$	0.997	0.994	0.999	0.998	0.996	0.994
Algorithm 3 ($k$-median) $\tau =\frac{4}{3}$	0.996	0.999	1.000	1.000	1.000	1.000
Algorithm 3 ($k$-median) $\tau =\frac{5}{3}$	0.995	1.000	1.000	1.000	1.000	1.000
Algorithm 3 ($k$-median) $\tau =2$	1.000	1.000	1.000	1.000	1.000	1.000
Algorithm 4 ($k$-median)	None	None	None	1.000	1.000	1.000
$k$-means$--$ ($k$-median)	0.996	0.999	1.000	1.000	1.000	1.000
LocalSearch ($k$-median)	0.999	1.000	1.000	1.000	1.000	1.000
DataSummary ($k$-median)	0.872	0.912	0.968	0.984	0.992	0.991

Table 3. Table 3: Precision on real datasets

$datasets$	Shuttle	Kddcup	Covtype	Poking Hand
Algorithm 1 $\tau =1$	0.904	0.966	0.612	0.959
Algorithm 1 $\tau =\frac{4}{3}$	0.906	0.964	0.695	0.978
Algorithm 1 $\tau =\frac{5}{3}$	0.913	0.965	0.753	0.975
Algorithm 1 $\tau =2$	0.904	0.966	0.908	0.984
Algorithm 2	0.903	0.957	0.513	0.965
MK	0.903	0.978	0.502	0.968
Malkomes	0.933	0.959	0.754	0.957
DYW	0.896	0.960	0.804	0.986
Algorithm 3 $\tau =1$	0.886	0.946	0.823	0.993
Algorithm 3 $\tau =\frac{4}{3}$	0.883	0.947	0.900	0.991
Algorithm 3 $\tau =\frac{5}{3}$	0.883	0.947	0.916	0.990
Algorithm 3 $\tau =2$	0.886	0.948	0.897	0.991
Algorithm 4	0.906	0.958	0.807	0.986
$k$-means$--$	0.883	0.971	0.793	0.999
LocalSearch	0.894	0.958	0.795	0.973
DataSummary	0.889	0.950	0.764	0.988
Algorithm 3 ($k$-median) $\tau =1$	0.889	0.946	0.728	0.995
Algorithm 3 ($k$-median) $\tau =\frac{4}{3}$	0.886	0.946	0.802	0.990
Algorithm 3 ($k$-median) $\tau =\frac{5}{3}$	0.881	0.946	0.802	0.990
Algorithm 3 ($k$-median) $\tau =2$	0.887	0.949	0.862	0.989
Algorithm 4 ($k$-median)	0.917	0.966	0.777	0.994
$k$-means$--$ ($k$-median)	0.882	0.966	0.733	0.999
LocalSearch ($k$-median)	0.891	0.953	0.751	0.964
DataSummary ($k$-median)	0.882	0.966	0.734	0.999

Table 4. Table 4: Purity on real datasets

$datasets$	Shuttle	Kddcup	Covtype	Poking Hand
Algorithm 1 $\tau =1$	0.810	0.690	0.491	0.502
Algorithm 1 $\tau =\frac{4}{3}$	0.837	0.848	0.493	0.504
Algorithm 1 $\tau =\frac{5}{3}$	0.859	0.872	0.496	0.505
Algorithm 1 $\tau =2$	0.913	0.904	0.504	0.505
Algorithm 2	0.830	0.851	0.495	0.504
MK	0.790	0.579	0.513	0.501
Malkomes	0.789	0.632	0.498	0.500
DYW	0.790	0.580	0.492	0.501
Algorithm 3 $\tau =1$	0.793	0.579	0.490	0.508
Algorithm 3 $\tau =\frac{4}{3}$	0.790	0.580	0.494	0.507
Algorithm 3 $\tau =\frac{5}{3}$	0.803	0.582	0.504	0.505
Algorithm 3 $\tau =2$	0.797	0.582	0.508	0.505
Algorithm 4	0.793	0.579	0.490	0.501
$k$-means$--$	0.818	0.579	0.491	0.501
LocalSearch	0.790	0.579	0.498	0.511
DataSummary	0.832	0.591	0.488	0.504
Algorithm 3 ($k$-median) $\tau =1$	0.790	0.582	0.491	0.508
Algorithm 3 ($k$-median) $\tau =\frac{4}{3}$	0.798	0.579	0.493	0.505
Algorithm 3 ($k$-median) $\tau =\frac{5}{3}$	0.797	0.581	0.499	0.504
Algorithm 3 ($k$-median) $\tau =2$	0.790	0.582	0.508	0.505
Algorithm 4 ($k$-median)	0.789	0.579	0.492	0.501
$k$-means$--$ ($k$-median)	0.790	0.579	0.492	0.501
LocalSearch ($k$-median)	0.811	0.627	0.496	0.510
DataSummary ($k$-median)	0.829	0.602	0.489	0.502

Equations

$${\Delta }_{\infty }^{-z}(P,H)$$

$${\Delta }_1^{-z}(P,H)$$

$${\Delta }_2^{-z}(P,H)$$

$$|S\cap C_j^{\ast }|$$

$$\frac{|S\cap C_j^{\ast }|}{|S|}\approx \frac{|C_j^{\ast }|}{n}\text{ and }\frac{1}{|S\cap C_j^{\ast }|}\sum _{q\in S\cap C_j^{\ast }}||q-o_j^{\ast }|{|}^2\approx \frac{1}{|C_j^{\ast }|}\sum _{p\in C_j^{\ast }}||p-o_j^{\ast }|{|}^2.$$

$$\displaystyle\bigg{|}\frac{1}{|T|}\sum_{q\in T}||q-o^{*}_{j}||^{2}-\frac{1}{|C^{*}_{j}|}\sum_{p\in C^{*}_{j}}||p-o^{*}_{j}||^{2}\bigg{|}\leq\xi\mathcal{L}^{2}$$

$$\displaystyle\sum_{q\in S\cap C^{*}_{j}}||q-o^{*}_{j}||^{2}\leq(1+\delta)\frac{|S|}{n}\big{(}\sum_{p\in C^{*}_{j}}||p-o^{*}_{j}||^{2}+\xi|C^{*}_{j}|\mathcal{L}^{2}\big{)}$$

$$\displaystyle\bigg{|}\frac{1}{|S\cap C^{*}_{j}|}\sum_{q\in S\cap C^{*}_{j}}||q-o^{*}_{j}||^{2}-\frac{1}{|C^{*}_{j}|}\sum_{p\in C^{*}_{j}}||p-o^{*}_{j}||^{2}\bigg{|}\leq\xi\mathcal{L}^{2}$$

$$\sum _{q\in S\cap C_j^{\ast }}||q-o_j^{\ast }|{|}^2$$

$$Cost(S_{opt},O^{\ast })$$

$$||\tilde{q}-h_{{\tilde{j}}_q}|{|}^2\le ||\tilde{q}-h_{j_q}|{|}^2\le 2||\tilde{q}-q|{|}^2+2||q-h_{j_q}|{|}^2$$

$$\sum _{q\in S_{opt}}||\tilde{q}-h_{{\tilde{j}}_q}|{|}^2$$

$$⟹Cost({\tilde{S}}_{opt},H)$$

$$Cost(S_{opt},H)\le Cost(S,H)\le c\cdot W,$$

$$W\le (|S|-k^{\prime }){\Delta }_2^{-k^{\prime }}(S,O^{\ast }).$$

$${\Delta }_2^{-z}(P,H)$$

$$\le$$

$$=$$

$$|S_{in}\cap C_j^{\ast }|$$

$$(|S|-z^{\prime }){\Delta }_2^{-z^{\prime }}(S,O^{\ast })\le Cost(S_{opt},O^{\ast }).$$

$${\Delta }_2^{-z^{\prime }}(S,H)\le c\cdot {\Delta }_2^{-z^{\prime }}(S,O^{\ast })\le \frac{c}{|S|-z^{\prime }}Cost(S_{opt},O^{\ast }),$$

$$Cost(S_{in}\cap P_{opt},H)$$

$$Cost({\tilde{S}}_{in}^{\prime },H)$$

$$Cost(P_{opt},H)$$

$$Cost({\tilde{P}}_{opt},H)$$

$$\displaystyle Cost(P_{opt},H)\leq 2Cost(P_{opt},O^{*})+\frac{4n}{|S|}\frac{t}{(t-1)(1-\delta)}\big{(}Cost(S^{\prime}_{in},O^{*})+Cost(S^{\prime}_{in},H)\big{)}$$

$$||o_{j_1}^{\ast }-o_{j_2}^{\ast }||$$

$$||q_1-q_2||$$

$$||o_j^{\ast }-q||$$

$$\displaystyle\textbf{Pr}(\big{|}\sum^{|S|}_{i=1}y_{i}-\tau|S|\big{|}\leq\delta\tau|S|)\geq 1-2e^{-\frac{\delta^{2}\tau}{3}|S|}.$$

Full text

1 Introduction

Clustering is a fundamental topic that has many important applications in real world [30]. An important type of clustering problems is called “center-based clustering” including the well-known $k$-center/median/means clustering problems [3]. Center-based clustering problems can be defined in arbitrary metrics and Euclidean space $\mathbb{R}^{D}$. Usually, a center-based clustering problem aims to find $k$ cluster centers so as to minimize the induced clustering cost. For example, the $k$-center clustering problem is to minimize the maximum distance from the input data to the set of cluster centers [26, 22]; the $k$-median (means) clustering problem is to minimize the average (squared) distance instead [35, 34].

Real-world datasets often contain outliers that could seriously destroy the final clustering results [43, 8]. Clustering with outliers can be viewed as a generalization of the ordinary clustering problems; however, the presence of outliers makes the problems to be much more challenging. Charikar et al. [9] proposed a $3$-approximation algorithm for $k$-center clustering with outliers in arbitrary metrics. The time complexity of their algorithm is at least quadratic in data size, since it needs to read all the pairwise distances. A following streaming $(4+\epsilon)$-approximation algorithm was proposed by McCutchen and Khuller [38]. Chakrabarty et al. [7] showed a $2$-approximation algorithm for metric $k$-center clustering with outliers based on the LP relaxation techniques. Recently, Ding et al. [16] provided a greedy algorithm that yields a bi-criteria approximation (returning more than $k$ clusters) based on the idea of the Gonzalez’s $k$-center clustering algorithm [22]. Bădoiu et al. [4] showed a coreset based approach but having an exponential time complexity if $k$ is not a constant (a “coreset” is a small set of points that approximates the structure/shape of a much larger point set, and thus can be used to significantly reduce the time complexities for many optimization problems [18]). The coresets for instances in doubling metrics were studied in [16, 6, 27].

For $k$-median/means clustering with outliers, the algorithms with provable guarantees [13, 32, 21] are difficult to implement due to their high complexities. Several heuristic algorithms without provable guarantee have been studied before [11, 42]. By using the local search method, Gupta et al.[25] provided a $274$-approximation algorithm for $k$-means clustering with outliers; they also showed that the well known $k$-means++ method [2] can be used as a coreset approach to reduce the complexity. Very recently, Im et al.[28] provided a method for constructing the coreset of $k$-means clustering with outliers by combining $k$-means++ and uniform sampling. Partly inspired by the successive sampling method of [39], Chen et al. [12] proposed a novel summary construction algorithm to reduce input data size.

Moreover, due to the rapid increase of data volumes in real world, a number of communication efficient distributed algorithms for $k$-center clustering with outliers [36, 23, 6, 33] and $k$-median/means clustering with outliers [23, 33, 12] were proposed in recent years.

1.1 Existing Sampling Methods and Our Main Results

As mentioned in above, existing algorithms for clustering with outliers often have high complexities (e.g., quadratic complexity). Therefore, several sampling methods have been studied for reducing the complexities. Namely, we take a small sample (uniformly or non-uniformly) from input and run an existing approximation algorithm on the sample. The non-uniform sampling methods include the aforementioned greedy algorithm [16], $k$-means++ [25, 28], and successive sampling [12]. However, these approaches suffer several drawbacks in practice; for example, they need to read the input dataset in multiple passes with high computational complexities, or have to discard more than the pre-specified number of outliers. The sensitivity-based coreset method is also a popular non-uniform sampling approach for ordinary clustering problems [19]. Informally, each data point $p$ has the “sensitivity” $\phi(p)$ to measure its importance to the whole dataset; the coreset construction is a simple sampling procedure where each point is drawn i.i.d. proportional to its sensitivity. However, to the best of our knowledge, the sensitivity-based coreset approach is not quite ideal to handle outliers, as it is not easy to compute the sensitivities because each point $p$ could be inlier or outlier for different solutions.

Due to the simplicity, the idea of uniform sampling has attracted a lot of attention. We follow the usual definition of “uniform sampling” in the articles [40, 41, 14], where it means that we take a sample from the input independently and uniformly at random. Suppose $n$ is the input size and $z$ is the number of outliers. Charikar et al.[10] and Meyerson et al. [40] respectively provided uniform sampling approaches for reducing data size for clustering with outliers; Huang et al. [27] and Ding et al. [16] presented similar results for instance in Euclidean space. However, these methods usually suffer the following dilemma.

The dilemma for uniform sampling. Let $S$ be the uniform sample from the input. If we try to avoid sampling any outlier, the size of $S$ should not be too large; but a small $S$ is more likely to yield a large clustering error. If we keep $|S|$ large enough, it is necessary to have an accurate enough estimation on the number of sampled outliers in $S$; that means $|S|$ should be larger than some threshold depending on $n/z$ (and the dimensionality $D$ for instance in Euclidean space), which could be very large (e.g., $D$ could be high and $z$ could be much smaller than $n$). As the results proposed in [27, 16], $S$ should be at least an $\epsilon$-sample (or some other variants) with the size depending on the VC-dimension in Euclidean space and the ratio $n/z$. Moreover, since it is impossible to know the exact number of sampled outliers in $S$ (even if $|S|$ is large), the error on the number of discarded outliers seems to be inevitable in almost all the previous uniform sampling approaches [10, 40, 27, 16]; that is, to guarantee the desired clustering qualities, they need to discard more than $z$ outliers.

The uniform sampling method was also applied to design sub-linear time algorithms for ordinary $k$-median/means clustering (without outliers) problems [40, 41, 14, 29], and we notice that the sample sizes proposed in [40, 14] are independent of $n/z$ and $D$. Therefore, a natural question is that whether their results can be extended for the clustering with outliers problems; in other words, is it possible to remove the dependencies on the values $n/z$ and $D$ in the sample complexity of uniform sampling? Another key question is that whether we can discard exactly $z$ outliers when using uniform sampling.

Our contributions. Though the uniform sampling approach suffers the above dilemma in theory, it often achieves nice performance in practice even if the sample size is much smaller than the theoretical bounds proposed in [10, 40, 27, 16]. To explain this phenomenon, we propose a new and unified framework for analyzing the effectiveness of uniform sampling for $k$-center/median/means clustering with outliers. We show that the sample size can be independent of the ratio $n/z$ and the dimensionality $D$, under some reasonable assumption. If we only require to output cluster centers, our uniform sampling approach runs in sub-linear time that is independent of the input size111Obviously, if we require to output the clustering membership for each point, it needs at least linear time.. To further boost the success probability, we can take multiple samples and select the one yielding the smallest objective value by scanning the whole dataset in one-pass. More importantly, to the best of our knowledge, our method is the first uniform sampling approach that allows to discard exactly $z$ outliers.

In our framework, we consider the relation between two values: the lower bound of the sizes of the optimal clusters $\inf_{1\leq j\leq k}|C^{*}_{j}|$ and the number of outliers $z$ (the formal definitions will be shown in Section 1.2). If a cluster $C^{*}_{j}$ has size $\ll z$, then we can say that $C^{*}_{j}$ is not a “significant” cluster. In real applications, we may only have an estimation for the number of clusters $k$. Consequently, if there exists a cluster $C^{*}_{j}$ having size $|C^{*}_{j}|\ll z$, we can formulate the problem as a simpler $(k-1)$-center/median/means clustering with $z+|C^{*}_{j}|$ outliers instead. So we can assume that each cluster $C^{*}_{j}$ has a size comparable to $z$. If $\frac{\inf_{1\leq j\leq k}|C^{*}_{j}|}{z}=\Omega(1)$ (note that the ratio could be a value smaller than $1$, say $0.5$, in this case), our framework outputs $k+O(\log k)$ cluster centers that yield a $4$-approximation for $k$-center clustering with outliers; further, if $\inf_{1\leq j\leq k}|C^{*}_{j}|>z$, our framework returns exactly $k$ cluster centers that yield a $(c+2)$-approximation solution, if we run an existing $c$-approximation algorithm with $c\geq 1$ on the sample. The framework can also handle $k$-median/means clustering with outliers and yields similar results.

We should point out that Meyerson et al. [40] also considered the lower bound of the cluster sizes when designing their sub-linear time $k$-median clustering (without outliers) algorithm. However, it is challenging to directly extend their idea to handle the case with outliers (as explained in the aforementioned dilemma for uniform sampling), and thus we need to develop significantly new ideas in our algorithms design and analysis. Recently, Gupta [24] proposed a similar uniform sampling approach to handle $k$-means clustering with outliers. However, their analysis and results are quite different from ours. Also their assumption is stronger: it requires that each optimal cluster has size roughly $\Omega(\frac{z}{\gamma^{2}}\log k)$ (i.e., $\frac{\inf_{1\leq j\leq k}|C^{*}_{j}|}{z}=\Omega(\frac{\log k}{\gamma^{2}})$) where $\gamma$ is a small parameter in $(0,1)$.

1.2 Preliminaries

Let the input be a point set $P\subset\mathbb{R}^{D}$ with $|P|=n$. Given a set of points $H\subset\mathbb{R}^{D}$ and a positive integer $z<n$, we define the following notations.

[TABLE]

where $dist(p,H)=\min_{q\in H}||p-q||$ and $||p-q||$ denotes the distance between $p$ and $q$.

Definition 1 ($k$-Center/Median/Means Clustering with Outliers)

Given a set $P$ of $n$ points in $\mathbb{R}^{D}$ with two positive integers $k$ and $z<n$, the problem of $k$-center (resp., $k$-median, $k$-means) clustering with outliers is to find $k$ cluster centers $C=\{c_{1},\cdots,c_{k}\}\subset\mathbb{R}^{D}$, such that the objective function $\Delta^{-z}_{\infty}(P,C)$ (resp., $\Delta^{-z}_{1}(P,C)$, $\Delta^{-z}_{2}(P,C)$) is minimized.

The definition can be easily modified for arbitrary metric space $(X,d)$, where $X$ contains $n$ vertices and $d(\cdot,\cdot)$ is the distance function: the Euclidean distance “$||p-q||$” is replaced by $d(p,q)$; the cluster centers $\{c_{1},\cdots,c_{k}\}$ should be chosen from $X$. In this paper, we always use $P_{opt}$, a subset of $P$ with size $n-z$, to denote the subset yielding the optimal solution with respect to the objective functions in Definition 1. Also, let $\{C^{*}_{1},\cdots,C^{*}_{k}\}$ be the $k$ optimal clusters forming $P_{opt}$.

As mentioned in Section 1.1, it is rational to assume that $\inf_{1\leq j\leq k}|C^{*}_{j}|$ is not far smaller than $z$. To formally state this assumption, we introduce the following definition.

Definition 2 ($(\epsilon_{1},\epsilon_{2})$-Significant Instance)

Let $\epsilon_{1},\epsilon_{2}>0$. Given an instance of $k$-center (resp., $k$-median, $k$-means) clustering with outliers as described in Definition 1, if $\inf_{1\leq j\leq k}|C^{*}_{j}|\geq\frac{\epsilon_{1}}{k}n$ and $z=\frac{\epsilon_{2}}{k}n$, we say that it is an $(\epsilon_{1},\epsilon_{2})$-significant instance.

Obviously, since $\sum^{k}_{j=1}|C^{*}_{j}|<n$, $\epsilon_{1}$ should be smaller than $1$. In Definition 2, we do not say “$\inf_{1\leq j\leq k}|C^{*}_{j}|=\frac{\epsilon_{1}}{k}n$”, since we may not be able to obtain the exact value of $\inf_{1\leq j\leq k}|C^{*}_{j}|$; instead, we may only have a lower bound of $\inf_{1\leq j\leq k}|C^{*}_{j}|$. The ratio $\frac{\epsilon_{1}}{\epsilon_{2}}$ ($\leq\frac{\inf_{1\leq j\leq k}|C^{*}_{j}|}{z}$) reveals the “significance” of the clusters to outliers; the higher the ratio is, the more significant the clusters to outliers will be.

The rest of the paper is organized as follows. To help our analysis, we present two implications of Definition 2 in Section 2. Then we introduce our uniform sampling algorithms for $k$-center clustering with outliers and $k$-median/means clustering with outliers in Section 3 and 4, respectively. We also explain that how to boost the success probability of our framework and how to further determine the clustering memberships of data points in Section 5. Finally, we present our experimental results in Section 7.

2 Implications of Significant Instance

Lemma 1

Given an $(\epsilon_{1},\epsilon_{2})$-significant instance $P$ as described in Definition 2, one selects a set $S$ of points from $P$ uniformly at random. Let $\eta,\delta\in(0,1)$. (i) If $|S|\geq\frac{k}{\epsilon_{1}}\log\frac{k}{\eta}$, with probability at least $1-\eta$, $S\cap C^{*}_{j}\neq\emptyset$ for any $1\leq j\leq k$. (ii) If $|S|\geq\frac{3k}{\delta^{2}\epsilon_{1}}\log\frac{2k}{\eta}$, with probability at least $1-\eta$, $|S\cap C^{*}_{j}|\in(1\pm\delta)\frac{|C^{*}_{j}|}{n}|S|$ for any $1\leq j\leq k$.

Lemma 1 can be obtained by using the Chernoff bound [1], and we leave the proof to Section 9. Moreover, we know that the expected number of outliers contained in the sample $S$ is $\frac{\epsilon_{2}}{k}|S|$. So we immediately have the following result by using the Markov’s inequality.

Lemma 2

Given an $(\epsilon_{1},\epsilon_{2})$-significant instance $P$ as described in Definition 2, one selects a set $S$ of points from $P$ uniformly at random. Let $\eta\in(0,1)$. With probability at least $1-\eta$, \big{|}S\setminus P_{opt}\big{|}\leq\frac{\epsilon_{2}}{k\eta}|S|.

3 Uniform Sampling for $k$-Center Clustering with Outliers

Let $r_{opt}$ be the optimal radius of the instance $P$, i.e., each optimal cluster $C^{*}_{j}$ is covered by a ball with radius $r_{opt}$. For any point $p\in\mathbb{R}^{D}$ and any value $r\geq 0$, we use $Ball(p,r)$ to denote the ball centered at $p$ with radius $r$. We present Algorithm 1 and 2 for the $k$-center clustering with outliers problem, and prove their clustering qualities in Theorem 3.1 and 3.2 respectively. We only focus on the problem in Euclidean space due to the space limit, but the results also hold for abstract metric space by using the same idea.

Theorem 3.1

In Algorithm 1, the size $|H|=k+\frac{1}{\eta}\frac{\epsilon_{2}}{\epsilon_{1}}\log\frac{k}{\eta}$. Also, with probability at least $(1-\eta)^{2}$, $\Delta^{-z}_{\infty}(P,H)\leq 4r_{opt}$.

Remark 1

(i) If we assume both $\frac{\epsilon_{2}}{\epsilon_{1}}$ and $\frac{1}{\eta}$ are $O(1)$, Theorem 3.1 indicates that Algorithm 1 returns $k+O(\log k)$ cluster centers. $\frac{\epsilon_{2}}{\epsilon_{1}}=O(1)$ means that $\frac{\inf_{1\leq j\leq k}|C^{*}_{j}|}{z}=\Omega(1)$, that is, the number of outliers is not significantly larger than the size of the smallest cluster. The running time of the subroutine algorithm [22] in Step 2 is $(k+k^{\prime})|S|D=O(\frac{k^{2}}{\epsilon_{1}}(\log k)D)$, which is independent of the input size $n$. (ii) In Section 6, we present an example to show that the value of $k^{\prime}$ cannot be further reduced. That is, the clustering quality could be arbitrarily bad if we run $(k+k^{\prime\prime})$-center clustering on $S$ with $k^{\prime\prime}<k^{\prime}$.

Proof

(of Theorem 3.1) First, it is straightforward to know that $|H|=k+k^{\prime}=k+\frac{1}{\eta}\frac{\epsilon_{2}}{\epsilon_{1}}\log\frac{k}{\eta}$. Below, we assume that the sample $S$ contains at least one point from each $C^{*}_{j}$, and at most $k^{\prime}=\frac{\epsilon_{2}}{k\eta}|S|$ points from $P\setminus P_{opt}$ (these events happen with probability at least $(1-\eta)^{2}$ due to Lemma 1 and 2).

Since the sample $S$ contains at most $k^{\prime}$ points from $P\setminus P_{opt}$ and $P_{opt}$ can be covered by $k$ balls with radius $r_{opt}$, we know that $S$ can be covered by $k+k^{\prime}$ balls with radius $r_{opt}$. Thus, if we perform the $2$-approximation $(k+k^{\prime})$-center clustering algorithm [22] on $S$, the resulting balls should have radius no larger than $2r_{opt}$. Let $H=\{h_{1},\cdots,h_{k+k^{\prime}}\}$ and $\mathbb{B}_{S}=\{Ball(h_{l},r)\mid 1\leq l\leq k+k^{\prime}\}$ be those balls covering $S$ with $r\leq 2r_{opt}$. Also, for each $1\leq j\leq k$, since $S\cap C^{*}_{j}\neq\emptyset$, there exists one ball of $\mathbb{B}_{S}$, say $Ball(h_{l_{j}},r)$, covers at least one point, say $p_{j}$, from $C^{*}_{j}$. For any point $p\in C^{*}_{j}$, we have $||p-p_{j}||\leq 2r_{opt}$ (by the triangle inequality) and $||p_{j}-h_{l_{j}}||\leq r\leq 2r_{opt}$; therefore, $||p-h_{l_{j}}||\leq||p-p_{j}||+||p_{j}-h_{l_{j}}||\leq 4r_{opt}$. See Figure 1. Overall, $P_{opt}=\cup^{k}_{j=1}C^{*}_{j}$ is covered by the balls $\cup^{k+k^{\prime}}_{l=1}Ball(h_{l},4r_{opt})$, i.e., $\Delta^{-z}_{\infty}(P,H)\leq 4r_{opt}$. ∎

Theorem 3.2

If $\frac{\epsilon_{1}}{\epsilon_{2}}>\frac{1}{\eta(1-\delta)}$, with probability at least $(1-\eta)^{2}$, Algorithm 2 returns $k$ cluster centers achieving a $(c+2)$-approximation for $k$-center clustering with $z$ outliers, i.e., $\Delta^{-z}_{\infty}(P,H)\leq(c+2)r_{opt}$.

Remark 2

For example, if we set $\eta=\delta=1/2$, the algorithm works for any instance with $\frac{\epsilon_{1}}{\epsilon_{2}}>\frac{1}{\eta(1-\delta)}=4$. Actually, as long as $\frac{\epsilon_{1}}{\epsilon_{2}}>1$ (i.e., $\inf_{1\leq j\leq k}|C^{*}_{j}|>z$), we can always find the appropriate values for $\eta$ and $\delta$ to satisfy $\frac{\epsilon_{1}}{\epsilon_{2}}>\frac{1}{\eta(1-\delta)}$. For example, we can set $\eta>\sqrt{\frac{\epsilon_{2}}{\epsilon_{1}}}$ and $\delta<1-\sqrt{\frac{\epsilon_{2}}{\epsilon_{1}}}$; obviously, if $\frac{\epsilon_{1}}{\epsilon_{2}}$ is close to $1$, the success probability $(1-\eta)^{2}$ could be small (we will show that how to boost the success probability in Section 5). The running time depends on the complexity of the subroutine $c$-approximation algorithm used in Step 2. For example, the algorithm of [9] takes O\big{(}|S|^{2}D+k|S|^{2}\log|S|\big{)} time in $\mathbb{R}^{D}$.

Proof

(of Theorem 3.2) We assume that $|S\cap C^{*}_{j}|\in(1\pm\delta)\frac{|C^{*}_{j}|}{n}|S|$ for each $C^{*}_{j}$, and $S$ has at most $z^{\prime}=\frac{\epsilon_{2}}{k\eta}|S|$ points from $P\setminus P_{opt}$ (these events happen with probability at least $(1-\eta)^{2}$ due to Lemma 1 and 2).

Let $\mathbb{B}_{S}=\{Ball(h_{l},r)\mid 1\leq l\leq k\}$ be the set of $k$ balls returned in Step 2 of Algorithm 2. Since $S\cap P_{opt}$ can be covered by $k$ balls with radius $r_{opt}$ and $|S\setminus P_{opt}|\leq z^{\prime}$, the optimal radius for the instance $S$ with $z^{\prime}$ outliers should be at most $r_{opt}$. Consequently, $r\leq cr_{opt}$. Moreover,

[TABLE]

for any $1\leq j\leq k$, where the last inequality comes from $\frac{\epsilon_{1}}{\epsilon_{2}}>\frac{1}{\eta(1-\delta)}$. Thus, if we perform $k$-center clustering with $z^{\prime}$ outliers on $S$, the resulting $k$ balls must cover at least one point from each $C^{*}_{j}$ (since $|S\cap C^{*}_{j}|>z^{\prime}$ from (4)). Through a similar manner as the proof of Theorem 3.1, we know that $P_{opt}=\cup^{k}_{j=1}C^{*}_{j}$ is covered by the balls $\cup^{k}_{l=1}Ball(h_{l},r+2r_{opt})$, i.e., $\Delta^{-z}_{\infty}(P,H)\leq r+2r_{opt}\leq(c+2)r_{opt}$. ∎

4 Uniform Sampling for $k$-Median/Means Clustering with Outliers

For the problem of $k$-means clustering with outliers, we apply the similar ideas as Algorithm 1 and 2 (see Algorithm 3 and 4). However, the analyses are more complicated here. For ease of understanding, we present our high-level idea first. Also, due to the space limit, we show the extensions for $k$-median clustering with outliers and their counterparts in arbitrary metric space in Section 10.

High-level idea. Let $S$ be a large enough random sample from $P$. Denote by $O^{*}=\{o^{*}_{1},\cdots,o^{*}_{k}\}$ the mean points of $\{C^{*}_{1},\cdots,C^{*}_{k}\}$, respectively. We first show that $S\cap C^{*}_{j}$ can well approximate $C^{*}_{j}$ for each $1\leq j\leq k$. Informally,

[TABLE]

We also define a transformation on $P_{opt}$ to help our analysis.

Definition 3 (Star Shaped Transformation)

For each point in $C^{*}_{j}$, we translate it to $o^{*}_{j}$; overall, we generate a new set of $n-z$ points located at $\{o^{*}_{1},\cdots o^{*}_{k}\}$, where each $o^{*}_{j}$ has $|C^{*}_{j}|$ overlapping points. For any point $p\in C^{*}_{j}$ with $1\leq j\leq k$, denote by $\tilde{p}$ its transformed point; for any $U\subseteq P_{opt}$, denote by $\tilde{U}$ its transformed point set.

Since the transformation forms $k$ “stars” (see Figure. 2), we call it “star shaped transformation”. Let $S_{opt}=S\cap P_{opt}$. By using (5), we can prove that the clustering costs of $\tilde{P}_{opt}$ and $\tilde{S}_{opt}$ are close (after the normalization) for any given set of cluster centers. Let $H$ be the $k+k^{\prime}$ cluster centers returned by Algorithm 3. Then, we can use $\tilde{S}_{opt}$ and $\tilde{P}_{opt}$ as the “bridges” between $S$ and $P$, so as to prove that $H$ yields an approximate solution for the instance $P$, i.e., $\Delta^{-z}_{2}(P,H)$ is bounded.

In Algorithm 4, we run $k$-means with $z^{\prime}$ outliers algorithm on the sample $S$ and return $k$ (rather than $k+k^{\prime}$) cluster centers. So we need to modify the above idea to analyze the quality. Let $S_{in}$ be the set of $|S|-z^{\prime}$ inliers of $S$ obtained in Step 2. If the ratio $\frac{\epsilon_{1}}{\epsilon_{2}}$ is large enough, we can prove that $|S_{in}\cap C^{*}_{j}|\approx|S\cap C^{*}_{j}|$ for each $1\leq j\leq k$. Therefore, we can replace “$S$” by “$S_{in}$” in (5) and prove a similar quality guarantee for Algorithm 4.

In Theorem 4.1 and 4.2, $\mathcal{L}$ denotes the maximum diameter of the $k$ clusters $C^{*}_{1},\cdots,C^{*}_{k}$, i.e., \mathcal{L}$$=\max_{1\leq j\leq k}\max_{p,q\in C^{*}_{j}} $||p-q||$. Actually, our result can be viewed as an extension of the sub-linear time $k$-median/means clustering algorithms [40, 41, 14] to the case with outliers. We also want to emphasize that the additive error is unavoidable even for the vanilla case (without outliers), if we require the sample complexity to be independent of the input size [41, 14]; the $k$-median clustering with outliers algorithm proposed by Meyerson et al. [40] does not yield an additive error, but it needs to discard more than $16z$ outliers and the sample size depends on the ratio $n/z$. We place the full proofs of Theorem 4.1 and 4.2 in Section 4.1 and 4.2 respectively.

Theorem 4.1

With probability at least $(1-\eta)^{3}$, the set of cluster centers $H$ returned by Algorithm 3 results in a clustering cost $\Delta^{-z}_{2}(P,H)$ at most $\alpha\Delta^{-z}_{2}(P,O^{*})+\beta\xi\mathcal{L}^{2}$, where \alpha=\big{(}2+(4+4c)\frac{1+\delta}{1-\delta}\big{)} and $\beta=(4+4c)\frac{1+\delta}{1-\delta}$.

Remark 3

(i) In Step 2 of Algorithm 3, we can apply an $O(1)$-approximation $k$-means algorithm (e.g., [31]). If we assume $1/\delta$ and $1/\eta$ are fixed constants, then the sample size $|S|=O(\frac{k}{\xi^{2}\epsilon_{1}}\log k)$, and both the factors $\alpha$ and $\beta$ are $O(1)$, i.e., $\Delta^{-z}_{2}(P,H)\leq O(1)\Delta^{-z}_{2}(P,O^{*})+O(1)\xi\mathcal{L}^{2}$; moreover, the number of returned cluster centers $|H|=k+O(\frac{\log k}{\xi^{2}})$ if $\frac{\epsilon_{1}}{\epsilon_{2}}=\Omega(1)$. (ii) Similar to Theorem 3.1, we use the same example in Section 6 to show that the value of $k^{\prime}$ cannot be further reduced in Algorithm 3.

Theorem 4.2

Assume $t=\eta(1-\delta)\frac{\epsilon_{1}}{\epsilon_{2}}>1$. With probability at least $(1-\eta)^{3}$, the set of cluster centers $H$ returned by Algorithm 4 results in a clustering cost $\Delta^{-z}_{2}(P,H)$ at most $\alpha\Delta^{-z}_{2}(P,O^{*})+\beta\xi\mathcal{L}^{2}$, where \alpha=\big{(}2+(4+4c)\frac{t}{t-1}\frac{1+\delta}{1-\delta}\big{)} and $\beta=(4+4c)\frac{t}{t-1}\frac{1+\delta}{1-\delta}$.

Similar to Theorem 3.2, as long as $\epsilon_{1}/\epsilon_{2}>1$, we can set $\eta>\sqrt{\frac{\epsilon_{2}}{\epsilon_{1}}}$ and $\delta<1-\sqrt{\frac{\epsilon_{2}}{\epsilon_{1}}}$ to keep $t>1$.

4.1 Proof of Theorem 4.1

The following lemma can be obtained via the Hoeffding’s inequality (each $||q-o^{*}_{j}||^{2}$ can be viewed as a random variable between [math] and $\mathcal{L}^{2}$) [1].

Lemma 3

We fix a cluster $C^{*}_{j}$. Given $\eta,\xi\in(0,1)$, if one uniformly selects a set $T$ of $\frac{1}{2\xi^{2}}\log\frac{2}{\eta}$ or more points at random from $C^{*}_{j}$,

[TABLE]

with probability at least $1-\eta$.

Lemma 4

If one uniformly selects a set $S$ of $\max\{\frac{3k}{\delta^{2}\epsilon_{1}}\log\frac{2k}{\eta},\frac{k}{2\xi^{2}\epsilon_{1}(1-\delta)}\log\frac{2k}{\eta}\}$ points at random from $P$,

[TABLE]

for $1\leq j\leq k$, with probability at least $(1-\eta)^{2}$.

Proof

Suppose $|S|=\max\{\frac{3k}{\delta^{2}\epsilon_{1}}\log\frac{2k}{\eta},\frac{k}{2\xi^{2}\epsilon_{1}(1-\delta)}\log\frac{2k}{\eta}\}$. According to Lemma 1, $|S|\geq\frac{3k}{\delta^{2}\epsilon_{1}}\log\frac{2k}{\eta}$ indicates that $|S\cap C^{*}_{j}|\geq(1-\delta)\frac{|C^{*}_{j}|}{n}|S|\geq(1-\delta)\frac{\epsilon_{1}}{k}|S|$ for each $1\leq j\leq k$; further, $|S|\geq\frac{k}{2\xi^{2}\epsilon_{1}(1-\delta)}\log\frac{2k}{\eta}$ implies that $(1-\delta)\frac{\epsilon_{1}}{k}|S|\geq\frac{1}{2\xi^{2}}\log\frac{2k}{\eta}$. Therefore, $|S\cap C^{*}_{j}|\geq\frac{1}{2\xi^{2}}\log\frac{2k}{\eta}$ and by Lemma 3 we have

[TABLE]

with probability at least $1-\eta$ for each $1\leq j\leq k$ ($\eta$ is replaced by $\eta/k$ in Lemma 3 for taking the union bound). From (8) we know that

[TABLE]

where the second inequality comes from Lemma 1. So we complete the proof. ∎

We define a new notation that is used in the following lemmas. Given two point sets $X$ and $Y\subset\mathbb{R}^{D}$, we use $Cost(X,Y)$ to denote the clustering cost of $X$ by taking $Y$ as the cluster centers, i.e., $Cost(X,Y)=\sum_{q\in X}(dist(q,Y))^{2}$. Obviously, $\Delta^{-z}_{2}(P,H)=\frac{1}{n-z}Cost(P_{opt},H)$. Let $S_{opt}=S\cap P_{opt}$. Below, we prove the upper bounds of $Cost(S_{opt},O^{*})$, $Cost(\tilde{S}_{opt},H)$, and $Cost(\tilde{P}_{opt},H)$ respectively, and use these bounds to complete the proof of Theorem 4.1. For convenience, we always assume that the events mentioned in Lemma 1, 2, and 4 all happen so that we do not need to repeatedly state the success probabilities.

Lemma 5

Cost(S_{opt},O^{*})\leq(1+\delta)\frac{|S|}{n}(n-z)\big{(}\Delta^{-z}_{2}(P,O^{*})+\xi\mathcal{L}^{2}\big{)}.

Proof

First, we have

[TABLE]

by Lemma 4. Further, since $\sum^{k}_{j=1}\sum_{p\in C^{*}_{j}}||p-o^{*}_{j}||^{2}=(n-z)\Delta^{-z}_{2}(P,O^{*})$ and $\sum^{k}_{j=1}|C^{*}_{j}|=n-z$, we know that $Cost(S_{opt},O^{*})$ is no larger than (1+\delta)\frac{|S|}{n}(n-z)\big{(}\Delta^{-z}_{2}(P,O^{*})+\xi\mathcal{L}^{2}\big{)}. ∎

Lemma 6

$Cost(\tilde{S}_{opt},H)\leq(2+2c)Cost(S_{opt},O^{*})$.

Proof

We fix a point $q\in S_{opt}$, and assume that the nearest neighbors of $q$ and $\tilde{q}$ in $H$ are $h_{j_{q}}$ and $h_{\tilde{j}_{q}}$, respectively. Then, we have

[TABLE]

via the triangle inequality. Therefore,

[TABLE]

Moreover, since $S_{opt}\subseteq S$ (because $S_{opt}=S\cap P_{opt}$) and $H$ yields a $c$-approximate clustering cost of the $(k+k^{\prime})$-means clustering on $S$, we have

[TABLE]

where $W$ is the optimal clustering cost of $(k+k^{\prime})$-means clustering on $S$. Let $S^{\prime}$ be the $k^{\prime}$ farthest points of $S$ to $O^{*}$, then the set $O^{*}\cup S^{\prime}$ also forms a solution for $(k+k^{\prime})$-means clustering on $S$; namely, $S$ is partitioned into $k+k^{\prime}$ clusters where each point of $S^{\prime}$ is a cluster having a single point. Obviously, such a clustering yields a clustering cost $(|S|-k^{\prime})\Delta^{-k^{\prime}}_{2}(S,O^{*})$. Consequently,

[TABLE]

Also, Lemma 2 shows that $S$ contains at most $k^{\prime}$ points from $P\setminus P_{opt}$, i.e., $|S_{opt}|\geq|S|-k^{\prime}$. Thus, $Cost(S_{opt},O^{*})\geq(|S|-k^{\prime})\Delta^{-k^{\prime}}_{2}(S,O^{*})$. Together with (12), (13), and (14), we have $Cost(\tilde{S}_{opt},H)\leq(2+2c)Cost(S_{opt},O^{*})$. ∎

Lemma 7

$Cost(\tilde{P}_{opt},H)\leq\frac{1}{1-\delta}\frac{n}{|S|}Cost(\tilde{S}_{opt},H)$.

Proof

From the constructions of $\tilde{P}_{opt}$ and $\tilde{S}_{opt}$, we know that they are overlapping points locating at $\{o^{*}_{1},\cdots,o^{*}_{k}\}$. From Lemma 1, we know $|S\cap C^{*}_{j}|\geq(1-\delta)\frac{|C^{*}_{j}|}{n}|S|$, i.e., $|C^{*}_{j}|\leq\frac{1}{1-\delta}\frac{n}{|S|}|S\cap C^{*}_{j}|$ for $1\leq j\leq k$. Overall, we have Cost(\tilde{P}_{opt},H)=\sum^{k}_{j=1}|C^{*}_{j}|\big{(}dist(o^{*}_{j},H)\big{)}^{2} that is at most \frac{1}{1-\delta}\frac{n}{|S|}\sum^{k}_{j=1}|S\cap C^{*}_{j}|\big{(}dist(o^{*}_{j},H)\big{)}^{2}=\frac{1}{1-\delta}\frac{n}{|S|}Cost(\tilde{S}_{opt},H). ∎

Now, we are ready to prove Theorem 4.1. Note that $(n-z)\Delta^{-z}_{2}(P,H)$ actually is the $|H|$-means clustering cost of $P$ by removing the farthest $z$ points to $H$, and $|P_{opt}|=n-z$. So we have $(n-z)\Delta^{-z}_{2}(P,H)\leq Cost(P_{opt},H)$. Further, by using a similar manner of (12), we have $Cost(P_{opt},H)\leq 2Cost(P_{opt},O^{*})+2Cost(\tilde{P}_{opt},H)$. Therefore,

[TABLE]

where (15), (16), and (17) come from Lemma 7, 6, and 5 respectively, and (18) comes from the fact $Cost(P_{opt},O^{*})=(n-z)\Delta^{-z}_{2}(P,O^{*})$. The success probability $(1-\eta)^{3}$ comes from Lemma 4 and Lemma 2 (note that Lemma 4 already takes into account of the success probability of Lemma 1 ). Thus, we obtain Theorem 4.1.

4.2 Proof of Theorem 4.2

Suppose the $k$ clusters of $S$ obtained in Step 2 of Algorithm 4 are $S_{1},S_{2},\cdots,S_{k}$, and thus the inliers $S_{in}=\cup^{k}_{j=1}S_{j}$. In the following lemmas, we always assume that the events mentioned in Lemma 1, 2, and 4 all happen so that we do not need to repeatedly state the success probabilities.

Lemma 8

$\frac{|C^{*}_{j}|}{|C^{*}_{j}\cap S_{in}|}\leq\frac{n}{|S|}\frac{t}{(t-1)(1-\delta)}$* for each $1\leq j\leq k$.*

Proof

Since $z^{\prime}=\frac{1}{\eta}\frac{\epsilon_{2}}{k}|S|$ and $|S\cap C^{*}_{j}|\geq(1-\delta)\frac{|C^{*}_{j}|}{n}|S|$ for each $1\leq j\leq k$ (by Lemma 1), we have

[TABLE]

where the last inequality comes from $|C^{*}_{j}|\geq\frac{\epsilon_{1}}{k}n$. Thus $\frac{|C^{*}_{j}|}{|C^{*}_{j}\cap S_{in}|}\leq\frac{n}{|S|}\frac{t}{(t-1)(1-\delta)}$. ∎

Lemma 9

Cost(S_{in}\cap P_{opt},H)\leq(1+\delta)\frac{|S|}{n}\cdot c\cdot\big{(}Cost(P_{opt},O^{*})+(n-z)\cdot\xi\mathcal{L}^{2}\big{)}.

Proof

Since $z^{\prime}\geq|S\setminus P_{opt}|=|S|-|S_{opt}|$, we have

[TABLE]

Because $H$ is a $c$-approximation on $S$,

[TABLE]

where the last inequality comes from (20). Therefore,

[TABLE]

where the second and third inequalities comes from (21) and Lemma 5, respectively. ∎

Since $S_{in}\cap P_{opt}\subseteq S_{opt}$, we immediately have the following lemma via Lemma 5.

Lemma 10

Cost(S_{in}\cap P_{opt},O^{*})\leq(1+\delta)\frac{|S|}{n}(n-z)\big{(}\Delta^{-z}_{2}(P,O^{*})+\xi\mathcal{L}^{2}\big{)}.

For convenience, let $S^{\prime}_{in}=S_{in}\cap P_{opt}$. Using the same manner of (12), we have

[TABLE]

Also, because $Cost(\tilde{P}_{opt},H)=\sum^{k}_{j=1}|C^{*}_{j}|(dist(o^{*}_{j},H))^{2}$ and $Cost(\tilde{S}^{\prime}_{in},H)=\sum^{k}_{j=1}|C^{*}_{j}\cap S_{in}|(dist(o^{*}_{j},H))^{2}$, we have

[TABLE]

where the last inequality comes from Lemma 8. From (23), (24), and (25), we have

[TABLE]

Combining Lemma 9 and 10, we can obtain Theorem 4.2 from (26) by simple calculation.

5 Success Probability and Clustering Memberships

The parameter $\eta$ determines the success probabilities of our algorithms. In particular, as mentioned in Remark 2, we cannot set $\eta$ too small to guarantee “$\frac{\epsilon_{1}}{\epsilon_{2}}>\frac{1}{\eta(1-\delta)}$” in Algorithm 2 (and similarly in Algorithm 4). To satisfy this requirement, we need to set $\eta$ large enough and therefore the success probability could be low. In fact, we can run the algorithm multiple times so as to achieve a higher success probability; for example, if $\eta=0.8$ and we run the algorithm $50$ times, the success probability will be $1-(1-(1-0.8)^{2})^{50}\approx 87\%$. Suppose we run the algorithm (Algorithm  2 or 4) $m>1$ times and let $H_{1},\cdots,H_{m}$ be the set of output candidates. The remaining issue is that how to select the one achieving the smallest objective value among all the candidates.

A simple way is to directly scan the whole dataset in one-pass. When reading a point $p$ from $P$, we calculate its distance to all the candidates, i.e., $dist(p,H_{1}),\cdots,dist(p,H_{m})$; after scanning the whole dataset, we have calculated the clustering costs $\Delta^{-z}_{\infty}(P,H_{l})$ (resp., $\Delta^{-z}_{1}(P,H_{l})$ and $\Delta^{-z}_{2}(P,H_{l})$) for $1\leq l\leq m$ and return the best one. Moreover, a by-product of this procedure is that we can determine the clustering memberships of data points simultaneously. When calculating $dist(p,H_{l})$ for $1\leq l\leq m$, we record the index of its nearest cluster center in $H_{l}$; finally, we return the corresponding clustering memberships after selecting the best candidate.

We are aware of the sampling method proposed by Meyerson et al. [40] for estimating $k$-median clustering cost; but it will induce an error on the number of outliers for our clustering with outliers problems. As mentioned in Section 1.1, the sampling based ideas in [10, 27, 16] also have the same issue.

6 An Example for Theorem 3.1 and 4.1

We construct the following instance for $k$-center clustering with outliers. Let $P$ be an $(\epsilon_{1},\epsilon_{2})$-significant instance in $\mathbb{R}^{D}$, where each optimal cluster $C^{*}_{j}$ is a set of $|C^{*}_{j}|$ overlapping points located at its cluster center $o^{*}_{j}$ for $1\leq j\leq k$. Let $x>0$, and we assume

[TABLE]

Obviously, the optimal radius $r_{opt}$ of $P$ is equal to [math]. Suppose we obtain a sample $S$ satisfying $S\cap C^{*}_{j}\neq\emptyset$ for any $1\leq j\leq k$ and $|S\setminus P_{opt}|=\frac{1}{\eta}\frac{\epsilon_{2}}{k}|S|=k^{\prime}$. Given a number $k^{\prime\prime}<k^{\prime}$, we run $(k+k^{\prime\prime})$-center clustering on $S$. Since the points of $S$ take $k+k^{\prime}$ distinct locations in the space, any $(k+k^{\prime\prime})$-center clustering on $S$ will result in a radius (at least $x/2$) larger than [math]; thus the approximation ratio is equal to $+\infty$. So, the value of $k^{\prime}$ cannot be further reduced in Theorem 3.1. It is easy to see that this instance $P$ also can be used to show that $k^{\prime}$ should be at least $\frac{1}{\eta}\frac{\epsilon_{2}}{k}|S|$ in Theorem 4.1.

7 Experiments

All the experimental results were obtained on a Windows workstation with 2.8GHz Intel(R) Core(TM) i5-840 and 8GB main memory; the algorithms are implemented in Matlab R2018a. To evaluate the performance, we use several baseline algorithms including the non-uniform sampling approaches mentioned in Section 1.1 (we only consider the setting with single machine in this paper). For $k$-center clustering with outliers, we consider four existing algorithms: the $3$-approximation Charikar [9], the $(4+\epsilon)$-approximation MK [38], the $13$-approximation Malkomes [36], and the greedy algorithm DYW [16] (as the non-uniform sampling approach). The distributed algorithm Malkomes partitions the dataset into $m\geq 1$ parts and processes each part separately; to make a fair comparison, we set $m=1$ for Malkomes in our experiments. In our Algorithm 2, we apply MK as the subroutine in Step 2.

For $k$-means/median clustering with outliers, we consider the heuristic algorithm $k$-means$--$ [11] and two non-uniform sampling methods: the local search algorithm LocalSearch with $k$-means++ [25], and the recent data summary based algorithm DataSummary [12]. In our Algorithm 3 (resp., Algorithm 4), we apply the $k$-means++ [2] (resp., $k$-means$--$) as the subroutine in Step 2.

Datasets. We generate the synthetic datasets in $\mathbb{R}^{100}$, and set $n=10^{5}$, $z=2\%n$, and $k=8$. We randomly generate k points as the cluster centers inside a hypercube of side length $400$; around each center, we generate a cluster of points following a Gaussian distribution with standard deviation $\sqrt{1000}$; we keep the total number of points to be $n-z$; to study the performance of our algorithms with respect to the ratio $\frac{\epsilon_{1}}{\epsilon_{2}}$, we vary the size of the smallest cluster appropriately for each synthetic dataset; finally, we uniformly generate $z$ outliers at random outside the minimum enclosing balls of these $k$ clusters.

We choose $4$ real datasets from UCI machine learning repository [17]. Covertype has $6$ clusters with $575000$ points in $\mathbb{R}^{54}$; Kddcup has $3$ clusters with $494021$ points in $\mathbb{R}^{38}$; Poking Hand has $4$ clusters with $995000$ points in $\mathbb{R}^{10}$; Shuttle has $3$ clusters with $59000$ points in $\mathbb{R}^{9}$. Each dataset also has some tiny clusters with total size $<1\%$, and we view them as outliers. To keep the fraction of outliers to be $1\%$, we add extra outliers outside the enclosing balls of the clusters as we did for the synthetic datasets.

Settings. For each dataset, we run our algorithms and the baseline algorithms $20$ trials and report the average results. For our algorithms, it is not quite convenient to set the values for the parameters $(\eta,\xi,\delta)$ in practice. Instead, it is more intuitive to directly set the sample size $|S|$ and $k^{\prime}$ for Algorithm 1 and 3 (resp., $z^{\prime}$ for Algorithm 2 and 4); actually, we only need these two numbers $(|S|,k^{\prime})$ (resp., $(|S|,z^{\prime})$) to implement our algorithms. In our experiments on both synthetic and real datasets, we set the sample size $|S|$ to be $2\%n$; for completeness, we also investigate the stability with varying $|S|/n$ in another experiment below. Algorithm 1 and 3 both output $k+k^{\prime}$ cluster centers, and we define the ratio $\tau=\frac{k+k^{\prime}}{k}$. The value $k^{\prime}=\frac{1}{\eta}\frac{\epsilon_{2}}{k}|S|$ could be large. Though Section 6 indicates that $k^{\prime}$ cannot be reduced with respect to the worst case, we do not strictly follow this (overly conservative) theoretical value in our experiments. Instead, we keep the ratio $\tau$ to be $1,\frac{4}{3},\frac{5}{3}$, and $2$ (i.e., run the algorithms [22, 2] $k+k^{\prime}=\tau k$ steps). For Algorithm 2 (resp., Algorithm 4), we set $z^{\prime}=2\frac{\epsilon_{2}}{k}|S|$ that is $2$ times the expected number of outliers in $S$; we run the algorithm $10$ times for each instance and select the best candidate by scanning the whole dataset in one-pass as discussed in Section 5 (we count the running time of the whole process). Further, we conduct another two experiments below to observe the influence of $z^{\prime}$ and the stabilities of the results returned by Algorithm 2 and 4 (if we just run each of them by one time).

Objective value and running time. The obtained objective values of our and the baseline algorithms are shown in Figure 3; the running times are shown in Figure 4; due to the space limit, we leave the experimental results of $k$-median clustering with outliers to appendix (Section 11). For $k$-center clustering with outliers, our algorithms (Algorithm 2 and Algorithm 1 with $\tau=2$) and the four baseline algorithms achieve similar objective values for most of the instances (we run Algorithm 2 on the synthetic datasets with $\frac{\epsilon_{1}}{\epsilon_{2}}>1$ only; we do not run Charikar on the real datasets due to its high complexity). Moreover, the running times of our Algorithm 1 and 2 are significantly lower comparing with the baseline algorithms. For $k$-means clustering with outliers, Algorithm 3 (except for the setting with $\tau=1$) and Algorithm 4 can achieve the results close to the best of the three baseline algorithms. DataSummary and Algorithm 4 achieve comparable running times, but Algorithm 4 outperforms DataSummary with respect to the objective values on the synthetic datasets.

The sample size $|S|$. We also study the influence of the sample size $|S|$ on the experimental performances of our algorithms. We vary the size $|S|$ from $2\%n$ to $10\%n$ and run our algorithms on the synthetic datasets ($\tau=2$ for Algorithm 1 and 3). We show the results in Figure 5. We can see that their performances stay stable when varying $|S|$.

The influences of $z^{\prime}$ and $\epsilon_{1}/\epsilon_{2}$ for Algorithm 2 and 4. We study the influence of $z^{\prime}$ on the performances. We vary $z^{\prime}/\tilde{z}$ from $1.1$ to $2$, where $\tilde{z}=\frac{\epsilon_{2}}{k}|S|$ is the expected number of outliers in $S$. We also study the stability of the obtained result if we just run Algorithm 2 (resp., Algorithm 4) by one time. We run each of them $200$ times on the synthetic datasets and show the results including the average objective values and standard deviations. From Figure 6, we can see that the performances are quite stable when varying $z^{\prime}/\tilde{z}$ and $\epsilon_{1}/\epsilon_{2}$.

Precision and purity. To further evaluate our experimental results, we compute the measures precision and purity, which have been widely used before [37]. The precision is the proportion of the ground-truth outliers found by the algorithm (i.e., $\frac{|Out\cap Out_{truth}|}{|Out_{truth}|}$, where $Out$ is the set of returned outliers and $Out_{truth}$ is the set of ground-truth outliers). For each obtained cluster, we assign it to the ground-truth cluster which is most frequent in the obtained cluster, and the purity measures the accuracy of this assignment. Specifically, let $\{C_{1},C_{2},\cdots,C_{k}\}$ be the ground-truth clusters and $\{C^{\prime}_{1},C^{\prime}_{2},\cdots,C^{\prime}_{k}\}$ be the obtained clusters from the algorithm; the purity is equal to $\frac{1}{n-z}\sum^{k}_{j=1}\max_{1\leq l\leq k}|C^{\prime}_{j}\cap C_{l}|$. The experimental results suggest that our algorithms can achieve the precisions and the purities comparable to those of the baselines. Due to the space limit, we leave the details to Section 12.

8 Future Work

Following this work, an interesting question is that whether the significance criterion can be applied to analyze the performance of uniform sampling for other well-known optimization problems, such as PCA with outliers [5] and projective clustering with outliers [20].

9 Proof of Lemma 1

Lemma 1 can be directly obtained through the following claim (we need to replace $\eta$ by $\eta/k$ in Claim 1, for taking the union bound over all the $k$ clusters).

Claim 1

Let $U$ be a set of elements and $V\subseteq U$ with $\frac{|V|}{|U|}=\tau>0$. Given $\eta,\delta\in(0,1)$, one uniformly selects a set $S$ of elements from $U$ at random. (i) If $|S|\geq\frac{1}{\tau}\log\frac{1}{\eta}$, with probability at least $1-\eta$, $S$ contains at least one element from $V$. (ii) If $|S|\geq\frac{3}{\delta^{2}\tau}\log\frac{2}{\eta}$, with probability at least $1-\eta$, we have \big{|}|S\cap V|-\tau|S|\big{|}\leq\delta\tau|S|.

Proof

Actually, (i) is a folklore result having been presented in several papers before (such as [15]). Since each sampled element falls in $V$ with probability $\tau$, we know that the sample $S$ contains at least one element from $V$ with probability $1-(1-\tau)^{|S|}$. Therefore, if we want $1-(1-\tau)^{|S|}\geq 1-\eta$, $|S|$ should be at least $\frac{\log 1/\eta}{\log 1/(1-\tau)}\leq\frac{1}{\tau}\log\frac{1}{\eta}$.

(ii) can be proved by using the Chernoff bound [1]. Define $|S|$ random variables $\{y_{1},\cdots,y_{|S|}\}$: for each $1\leq i\leq|S|$, $y_{i}=1$ if the $i$-th sampled element falls in $V$, otherwise, $y_{i}=0$. So $E[y_{i}]=\tau$ for each $y_{i}$. As a consequence, we have

[TABLE]

If $|S|\geq\frac{3}{\delta^{2}\tau}\log\frac{2}{\eta}$, with probability at least $1-\eta$, \big{|}\sum^{|S|}_{i=1}y_{i}-\tau|S|\big{|}\leq\delta\tau|S| (i.e., \big{|}|S\cap V|-\tau|S|\big{|}\leq\delta\tau|S|). ∎

10 Extensions of Theorem 4.1 and 4.2

The results of Theorem 4.1 and 4.2 can be easily extended to $k$-median clustering with outliers in Euclidean space by using almost the same idea, where the only difference is that we can directly use triangle inequality in the proof (e.g., the inequality (11) is replaced by $||\tilde{q}-h_{\tilde{j}_{q}}||\leq||\tilde{q}-q||+||q-h_{j_{q}}||$); the coefficients $\alpha$ and $\beta$ are reduced to be \big{(}1+(1+c)\frac{1+\delta}{1-\delta}\big{)} and $(1+c)\frac{1+\delta}{1-\delta}$ (resp., \big{(}1+(1+c)\frac{t}{t-1}\frac{1+\delta}{1-\delta}\big{)} and $(1+c)\frac{t}{t-1}\frac{1+\delta}{1-\delta}$) in Theorem 4.1 (resp., Theorem 4.2), respectively.

To solve the metric $k$-median/means clustering with outliers problems for an instance $(X,d)$, we should keep in mind that the cluster centers can only be selected from the vertices of $X$. However, the optimal cluster centers $O^{*}=\{o^{*}_{1},\cdots,o^{*}_{k}\}$ may not be contained in the sample $S$, and thus we need to modify our analysis slightly. We observe that the sample $S$ contains a set $O^{\prime}$ of vertices close to $O^{*}$ with certain probability. Specifically, for each $1\leq j\leq k$, there exists a vertex $o^{\prime}_{j}\in O^{\prime}$ such that $d(o^{\prime}_{j},o^{*}_{j})\leq O(1)\times\frac{1}{|C^{*}_{j}|}\sum_{p\in C^{*}_{j}}d(p,o^{*}_{j})$ (or \big{(}d(o^{\prime}_{j},o^{*}_{j})\big{)}^{2}\leq O(1)\times\frac{1}{|C^{*}_{j}|}\sum_{p\in C^{*}_{j}}\big{(}d(p,o^{*}_{j})\big{)}^{2}) with constant probability (this claim can be easily proved by using the Markov’s inequality). Consequently, we can use $O^{\prime}$ to replace $O^{*}$ in our analysis, and achieve the similar results as Theorem 4.1 and 4.2.

11 The Experimental Results for $K$-median Clustering with Outliers

12 Precision and Purity