Machine Learning Friendly Set Version of Johnson-Lindenstrauss Lemma

TL;DR

This paper introduces a probabilistic approach to the Johnson-Lindenstrauss Lemma, enabling pre-determined target dimensions for data projection with high confidence, and demonstrates its effectiveness for clustering and k-means algorithms.

Contribution

It formulates a theorem for selecting target dimensions in JL transformations with probabilistic guarantees, improving applicability in data clustering and k-means analysis.

Findings

Probabilistic bounds for target dimension selection in JL transformations

Preservation of clustering quality in projected space

Conditions for clusterability transfer to the projected space

Abstract

In this paper we make a novel use of the Johnson-Lindenstrauss Lemma. The Lemma has an existential form saying that there exists a JL transformation $f$ of the data points into lower dimensional space such that all of them fall into predefined error range $\delta$. We formulate in this paper a theorem stating that we can choose the target dimensionality in a random projection type JL linear transformation in such a way that with probability $1-\epsilon$ all of them fall into predefined error range $\delta$ for any user-predefined failure probability $\epsilon$. This result is important for applications such a data clustering where we want to have a priori dimensionality reducing transformation instead of trying out a (large) number of them, as with traditional Johnson-Lindenstrauss Lemma. In particular, we take a closer look at the $k$-means algorithm and prove that a good solution in the projected space is also a good solution in the original space. Furthermore, under proper assumptions local optima in the original space are also ones in the projected space. We define also conditions for which clusterability property of the original space is transmitted to the projected space, so that special case algorithms for the original space are also applicable in the projected space.

Figures

Tables

Table 1. Table 1: Dependence of reduced dimensionality n ′ superscript 𝑛 ′ n^{\prime} on sample size m 𝑚 m . Other parameters fixed at ϵ italic-ϵ \epsilon =0.01 δ 𝛿 \delta =0.05 n 𝑛 n =5e+05.

$m$	$n^{\prime }$ explicit	$n^{\prime }$ implicit	explicit/implicit
10	15226	14209	1.07
20	17518	16389	1.07
50	20547	19191	1.07
100	22839	21269	1.07
200	25131	23323	1.08
500	28160	26016	1.08
1000	30452	28030	1.09
2000	32744	30027	1.09
5000	35773	32648	1.1
10000	38065	34609	1.1
20000	40357	36554	1.1
50000	43386	39097	1.11
1e+05	45678	41017	1.11
2e+05	47970	42910	1.12
5e+05	50999	45392	1.12
1e+06	53291	47250	1.13
2e+06	55582	49099	1.13
5e+06	58612	51515	1.14
1e+08	68516	59243	1.16
2e+07	63195	55127	1.15
5e+07	66225	57480	1.15
1e+08	68516	59243	1.16

Table 2. Table 2: Dependence of reduced dimensionality n ′ superscript 𝑛 ′ n^{\prime} on failure prob. ϵ italic-ϵ \epsilon . Other parameters fixed at m 𝑚 m =2e+06 δ 𝛿 \delta =0.05 n 𝑛 n =5e+05.

$ϵ$	$n^{\prime }$ explicit	$n^{\prime }$ implicit	explicit/implicit
0.1	51776	46020	1.13
0.05	52922	46955	1.13
0.02	54437	48180	1.13
0.01	55582	49099	1.13
0.005	56728	50014	1.13
0.002	58243	51221	1.14
0.001	59389	52134	1.14

Table 3. Table 3: Dependence of reduced dimensionality n ′ superscript 𝑛 ′ n^{\prime} on error range δ 𝛿 \delta . Other parameters fixed at m 𝑚 m =2e+06 ϵ italic-ϵ \epsilon =0.01 n 𝑛 n =5e+05.

$\delta$	$n^{\prime }$ explicit	$n^{\prime }$ implicit	explicit/implicit
0.5	712	697	1.02
0.4	1059	1032	1.03
0.3	1787	1745	1.02
0.2	3804	3692	1.03
0.1	14339	13640	1.05
0.09	17593	16631	1.06
0.08	22128	20742	1.07
0.07	28721	26604	1.08
0.06	38846	35329	1.1
0.05	55582	49099	1.13
0.04	86291	72387	1.19
0.03	152415	115298	1.32
0.02	340701	201059	1.69
0.01	1353858	1353859	1

Table 4. Table 4: Dependence of reduced dimensionality n ′ superscript 𝑛 ′ n^{\prime} on original dimensionality n 𝑛 n . Other parameters fixed at m 𝑚 m =2e+06 ϵ italic-ϵ \epsilon =0.01 δ 𝛿 \delta =0.05.

$n$	$n^{\prime }$ explicit	$n^{\prime }$ implicit	explicit/implicit
4e+05	55582	47891	1.16
5e+05	55582	49099	1.13
6e+05	55582	49933	1.11
7e+05	55582	50551	1.1
8e+05	55582	51025	1.09
9e+05	55582	51399	1.08
1e+06	55582	51703	1.08

Table 5. Table 5: Comparison of effort needed for k 𝑘 k -means under our dimensionality reduction approach and that of Dasgupta and Gupta [ 11 ] , depending on sample size m 𝑚 m . Other parameters fixed at ϵ italic-ϵ \epsilon = 0.01 δ 𝛿 \delta = 0.05 n 𝑛 n = 5e+05

$m$	$n^{\prime }$ explicit	$n^{\prime }$ implicit	Gupta $n^{\prime }$	Their Repetitions	Our $n^{\prime }$ to their $n^{\prime }$
10	15226	14209	3879	44	3.7
20	17518	16389	5046	90	3.3
50	20547	19191	6589	228	3
100	22839	21269	7757	459	2.8
200	25131	23323	8924	919	2.7
500	28160	26016	10467	2301	2.5
1000	30452	28030	11635	4603	2.5
2000	32744	30027	12802	9209	2.4
5000	35773	32648	14345	23024	2.3
10000	38065	34609	15513	46050	2.3
20000	40357	36554	16680	92102	2.2
50000	43386	39097	18223	230257	2.2
1e+05	45678	41017	19391	460515	2.2
2e+05	47970	42910	20558	921032	2.1
5e+05	50999	45392	22101	2302583	2.1
1e+06	53291	47250	23269	4605168	2.1
2e+06	55582	49099	24436	9210339	2.1
5e+06	58612	51515	25979	23025849	2
1e+08	68516	59243	31025	460517014	2
2e+07	63195	55127	28314	92103402	2
5e+07	66225	57480	29857	230258508	2
1e+08	68516	59243	31025	460517014	2

Table 6. Table 6: Comparison of effort needed for k 𝑘 k -means under our dimensionality reduction approach and that of Dasgupta and Gupta [ 11 ] , depending on failure prob. ϵ italic-ϵ \epsilon . Other parameters fixed at m= 2e+06 δ 𝛿 \delta = 0.05 n 𝑛 n = 5e+05

$ϵ$	$n^{\prime }$ explicit	$n^{\prime }$ implicit	Gupta $n^{\prime }$	Their Repetitions	Our $n^{\prime }$ to their $n^{\prime }$
0.1	51776	46020	24436	4605170	1.9
0.05	52922	46955	24436	5991464	2
0.02	54437	48180	24436	7824045	2
0.01	55582	49099	24436	9210339	2.1
0.005	56728	50014	24436	10596633	2.1
0.002	58243	51221	24436	12429214	2.1
0.001	59389	52134	24436	13815508	2.2

Table 7. Table 7: Comparison of effort needed for k 𝑘 k -means under our dimensionality reduction approach and that of Dasgupta and Gupta [ 11 ] , depending on error range δ 𝛿 \delta . Other parameters fixed at m= 2e+06 ϵ italic-ϵ \epsilon = 0.01 n 𝑛 n = 5e+05

$\delta$	$n^{\prime }$ explicit	$n^{\prime }$ implicit	Gupta $n^{\prime }$	Their Repetitions	Our $n^{\prime }$ to their $n^{\prime }$
0.5	712	697	465	9210339	1.5
0.4	1059	1032	605	9210339	1.8
0.3	1787	1745	922	9210339	1.9
0.2	3804	3692	1814	9210339	2.1
0.1	14339	13640	6449	9210339	2.2
0.09	17593	16631	7874	9210339	2.2
0.08	22128	20742	9857	9210339	2.2
0.07	28721	26604	12736	9210339	2.1
0.06	38846	35329	17150	9210339	2.1
0.05	55582	49099	24436	9210339	2.1
0.04	86291	72387	37783	9210339	2
0.03	152415	115298	66478	9210339	1.8
0.02	340701	201059	148048	9210339	1.4
0.01	1353858	1353859	586209	9210339	2.4

Table 8. Table 8: Comparison of effort needed for k 𝑘 k -means under our dimensionality reduction approach and that of Dasgupta and Gupta [ 11 ] , depending on original dimensionality n 𝑛 n . Other parameters fixed at m= 2e+06 ϵ italic-ϵ \epsilon = 0.01 δ 𝛿 \delta = 0.05

$n$	$n^{\prime }$ explicit	$n^{\prime }$ implicit	Gupta $n^{\prime }$	Their Repetitions	Our $n^{\prime }$ to their $n^{\prime }$
4e+05	55582	47891	24436	9210339	2
5e+05	55582	49099	24436	9210339	2.1
6e+05	55582	49933	24436	9210339	2.1
7e+05	55582	50551	24436	9210339	2.1
8e+05	55582	51025	24436	9210339	2.1
9e+05	55582	51399	24436	9210339	2.2
1e+06	55582	51703	24436	9210339	2.2

Equations

$$(1-\delta )\Vert \mathbf{u}-\mathbf{v}{\Vert }^2\le \Vert f(\mathbf{u})-f(\mathbf{v}){\Vert }^2\le (1+\delta )\Vert \mathbf{u}-\mathbf{v}{\Vert }^2$$

$$\mathfrak{J}(Q,\mathfrak{C})=\sum _{i\in \mathfrak{Q}}\Vert {\mathbf{x}}_i-\mathbf{\mu }(\mathfrak{C}(i)){\Vert }^2$$

$$\mathfrak{J}(Q^{\prime },\mathfrak{C})=\sum _{i\in \mathfrak{Q}}\Vert {{\mathbf{x}}^{\prime }}_i-{\mathbf{\mu }}^{\prime }(\mathfrak{C}(i){\Vert }^2$$

$$n^{\prime }\ge 2\frac{-\ln ϵ+2\ln (m)}{-\ln (1+\delta )+\delta }$$

$$(1-\delta )\mathfrak{J}(Q,\mathfrak{C})\le \frac{n}{n^{\prime }}\mathfrak{J}(Q^{\prime },\mathfrak{C})\le (1+\delta )\mathfrak{J}(Q,\mathfrak{C})$$

$$\left(1-\frac{\delta }{1+\delta }\right)\mathfrak{J}(Q^{\prime },\mathfrak{C})\le \frac{n^{\prime }}{n}\mathfrak{J}(Q,\mathfrak{C})\le \left(1+\frac{\delta }{1-\delta }\right)\mathfrak{J}(Q^{\prime },\mathfrak{C})$$

$$\delta \le \frac{1-{\left(1-\frac{g}{2}\right)}^2}{{\left(1-\frac{g}{2}\right)}^2+(1+2p)}$$

$$\frac{\delta }{1-\delta }\le \frac{1-{\alpha }^2}{(1+2p)+{\alpha }^2}$$

$$\frac{n}{n^{\prime }}\mathfrak{J}(Q^{\prime },{\mathfrak{C}}_{\mathfrak{G}}^{\prime })\le (1+\delta )\mathfrak{J}(Q,{\mathfrak{C}}_{\mathfrak{G}})$$

$$Pr(\Vert {\mathbf{x}}^{\prime }{\Vert }^2\le \beta \frac{n^{\prime }}{n}\Vert \mathbf{x}{\Vert }^2)\le {\beta }^{\frac{n^{\prime }}{2}}{\left(1+\frac{n^{\prime }(1-\beta )}{n-n^{\prime }}\right)}^{\frac{n-n^{\prime }}{2}}$$

$$Pr(\Vert {\mathbf{x}}^{\prime }{\Vert }^2\ge \beta \frac{n^{\prime }}{n}\Vert \mathbf{x}{\Vert }^2)\le {\beta }^{\frac{n^{\prime }}{2}}{\left(1+\frac{n^{\prime }(1-\beta )}{n-n^{\prime }}\right)}^{\frac{n-n^{\prime }}{2}}$$

$$Pr\left((1-\delta )\Vert \mathbf{x}{\Vert }^2\le \frac{n}{n^{\prime }}\Vert {\mathbf{x}}^{\prime }{\Vert }^2\le (1+\delta )\Vert \mathbf{x}{\Vert }^2\right)$$

$$\ge 1-(1-\delta {)}^{\frac{n^{\prime }}{2}}{\left(1+\frac{n^{\prime }\delta }{n-n^{\prime }}\right)}^{\frac{n-n^{\prime }}{2}}$$

$$-(1+\delta {)}^{\frac{n^{\prime }}{2}}{\left(1-\frac{n^{\prime }\delta }{n-n^{\prime }}\right)}^{\frac{n-n^{\prime }}{2}}$$

$$Pr\left((1-\delta )\Vert \mathbf{x}{\Vert }^2\le \frac{n}{n^{\prime }}\Vert {\mathbf{x}}^{\prime }{\Vert }^2\le (1+\delta )\Vert \mathbf{x}{\Vert }^2\right)$$

$$\ge 1-2\max ((1-\delta {)}^{\frac{n^{\prime }}{2}}{\left(1+\frac{n^{\prime }\delta }{n-n^{\prime }}\right)}^{\frac{n-n^{\prime }}{2}},$$

$$(1+\delta {)}^{\frac{n^{\prime }}{2}}{\left(1-\frac{n^{\prime }\delta }{n-n^{\prime }}\right)}^{\frac{n-n^{\prime }}{2}})$$

$$=1-2\underset{{\delta }^{\ast }\in \{-\delta ,+\delta \}}{\max }\left({\left(1-{\delta }^{\ast }\right)}^{\frac{n^{\prime }}{2}}{\left(1+\frac{{\delta }^{\ast }{n}^{\prime }}{n-n^{\prime }}\right)}^{\frac{n-n^{\prime }}{2}}\right)$$

$$1-ϵ\le 1-\left(\genfrac{}{}{0ex}{}{m}{2}\right)\left(1-Pr\left((1-\delta )\Vert \mathbf{x}{\Vert }^2\le \frac{n}{n^{\prime }}\Vert {\mathbf{x}}^{\prime }{\Vert }^2\le (1+\delta )\Vert \mathbf{x}{\Vert }^2\right)\right)$$

$$ϵ\ge 2\left(\genfrac{}{}{0ex}{}{m}{2}\right)\underset{{\delta }^{\ast }\in \{-\delta ,+\delta \}}{\max }\left((1-{\delta }^{\ast }{)}^{\frac{n^{\prime }}{2}}{\left(1+\frac{{\delta }^{\ast }{n}^{\prime }}{n-n^{\prime }}\right)}^{\frac{n-n^{\prime }}{2}}\right)$$

$$\ln ϵ\ge$$

$$+\underset{{\delta }^{\ast }\in \{-\delta ,+\delta \}}{\max }\left(\frac{n^{\prime }}{2}\ln (1-{\delta }^{\ast })+\frac{(n-n^{\prime })}{2}\ln \left(1+\frac{{\delta }^{\ast }{n}^{\prime }}{n-n^{\prime }}\right)\right)$$

$$\ln ϵ-\ln (m(m-1))$$

$$\ge \underset{{\delta }^{\ast }\in \{-\delta ,+\delta \}}{\max }\left(\frac{n^{\prime }}{2}\ln (1-{\delta }^{\ast })+\frac{(n-n^{\prime })}{2}\ln \left(1+\frac{{\delta }^{\ast }{n}^{\prime }}{n-n^{\prime }}\right)\right)$$

$$\ln ϵ-\ln (m(m-1))\ge \underset{{\delta }^{\ast }\in \{-\delta ,+\delta \}}{\max }\left(\frac{n^{\prime }}{2}\ln (1-{\delta }^{\ast })+\frac{(n-n^{\prime })}{2}\frac{{\delta }^{\ast }{n}^{\prime }}{n-n^{\prime }}\right)$$

$$\ln ϵ-\ln (m(m-1))\ge \underset{{\delta }^{\ast }\in \{-\delta ,+\delta \}}{\max }\left(\frac{n^{\prime }}{2}\ln (1-{\delta }^{\ast })+\frac{1}{2}({\delta }^{\ast })n^{\prime }\right)=\frac{n^{\prime }}{2}\underset{{\delta }^{\ast }\in \{-\delta ,+\delta \}}{\max }\left(\ln (1-{\delta }^{\ast })+{\delta }^{\ast }\right)$$

$$\underset{{\delta }^{\ast }\in \{-\delta ,+\delta \}}{\max }\left(2\frac{\ln ϵ-\ln (m(m-1))}{\ln (1-{\delta }^{\ast })+{\delta }^{\ast }}\right)\le n^{\prime }$$

$$n^{\prime }\ge 2\frac{-\ln ϵ+2\ln (m)}{-\ln (1+\delta )+\delta }$$

$$n^{\prime }\ge (2+r)\frac{2\ln (m)}{-\ln (1+\delta )+\delta }$$

$$n^{\prime }\ge 2\frac{-\ln ϵ+2\ln (m)}{-\ln (1+\delta )+\delta }$$

$$(1-\delta )\Vert \mathbf{u}-\mathbf{v}{\Vert }^2\le \frac{n}{n^{\prime }}\Vert {\mathbf{u}}^{\prime }-{\mathbf{v}}^{\prime }{\Vert }^2\le (1+\delta )\Vert \mathbf{u}-\mathbf{v}{\Vert }^2$$

$$(1-\delta )\Vert {\mathbf{x}}_{\mathbf{i}}-\mathbf{\mu }(\mathfrak{C}(i)){\Vert }^2\le \frac{n}{n^{\prime }}\Vert {\mathbf{x}}_{\mathbf{i}}^{\prime }-{\mathbf{\mu }}^{\prime }(\mathfrak{C}(i)){\Vert }^2\le (1+\delta )\Vert {\mathbf{x}}_i-\mathbf{\mu }(\mathfrak{C}(i)){\Vert }^2$$

$$\Vert {\mathbf{x}}_i-\mathbf{\mu }(\mathfrak{C}(i)){\Vert }^2=\frac{1}{|\mathfrak{C}(i)|}\sum _{j\in \mathfrak{C}(i)}\Vert {\mathbf{x}}_{\mathbf{i}}-{\mathbf{x}}_{\mathbf{j}}{\Vert }^2$$

$$(1-\delta )\sum _{j\in \mathfrak{C}(i)}\Vert {\mathbf{x}}_i-{\mathbf{x}}_j{\Vert }^2\le \frac{n}{n^{\prime }}\sum _{j\in \mathfrak{C}(i)}\Vert {\mathbf{x}}_i^{\prime }-{\mathbf{x}}_j^{\prime }{\Vert }^2\le (1+\delta )\sum _{j\in \mathfrak{C}(i)}\Vert {\mathbf{x}}_i-{\mathbf{x}}_j{\Vert }^2$$

Full text

We formulate in this paper a theorem stating that we can choose the target dimensionality in a random projection type JL linear transformation in such a way that with probability $1-\epsilon$ all of them fall into predefined error range $\delta$ for any user-predefined failure probability $\epsilon$.

This result is important for applications such a data clustering where we want to have a priori dimensionality reducing transformation instead of trying out a (large) number of them, as with traditional Johnson-Lindenstrauss Lemma. In particular, we take a closer look at the $k$-means algorithm and prove that a good solution in the projected space is also a good solution in the original space. Furthermore, under proper assumptions local optima in the original space are also ones in the projected space. We define also conditions for which clusterability property of the original space is transmitted to the projected space, so that special case algorithms for the original space are also applicable in the projected space.

Keywords:

Johnson-Lindenstrauss Lemma, random projection, sample distortion, dimensionality reduction, linear JL transform, $k$-means algorithm, clusterability retention,

1 Introduction

Dimensionality reduction plays an important role in many areas of data processing, and especially in machine learning (cluster analysis, classifier learning, model validation, data visualisation etc.).

Usually it is associated with manifold learning, that is a belief that the data lie in fact in a low dimensional subspace that needs to be identified and the data projected onto it so that the number of degrees of freedom is reduced and as a consequence also sample sizes can be smaller without loss of reliability. Techniques like reduced $k$-means [17], PCA (Principal Component Analysis), Kernel PCA, LLE (Locally Linear Embedding), LEM (Laplacian Eigenmaps), MDS (Metric Multidimensional Scaling), Isomap, SDE (Semidefinite Embedding), just to mention a few.

But there exists still another possibility of approaching the dimensionality reduction problems, in particular when such intrinsic subspace where data is located cannot be identified. The problem of choice of the subspace has been surpassed by several authors by so-called random projection, applicable in particularly highly dimensional spaces (tens of thousands of dimensions) and correspondingly large data sets (of at least hundreds of points).

The starting point here is the Johnson-Lindenstrauss Lemma [13]. Roughly speaking it states that there exists a linear111JL Lemma speaks about a general transformation, but many researchers look just for linear ones. mapping from a higher dimensional space into a sufficiently high dimensional subspace that will preserve approximately the distances between points, as needed e.g. by $k$-means algorithm [4].

To be more formal consider a set $\mathfrak{Q}$ of $m$ objects $\mathfrak{Q}=\{1,\dots,m\}$. An object $i\in\mathfrak{Q}$ may have a representation $\mathbf{x_{i}}\in\mathbb{R}^{n}$. Then the set of these representations will be denoted by $Q$. An object $i\in\mathfrak{Q}$ may have a representation $\mathbf{x^{\prime}_{i}}\in\mathbb{R}^{n^{\prime}}$, in a different space. Then the set of these representations will be denoted by $Q^{\prime}$.

With this notation let us state:

Theorem 1**.**

*(Johnson-Lindenstrauss) Let $\delta\in(0,\frac{1}{2})$. Let $\mathfrak{Q}$ be a set of $m$ objects and $Q$

• a set of points representing them in $\mathbb{R}^{n}$, and let $n^{\prime}\geq\frac{C\ln m}{\delta^{2}}$, where $C$ is a sufficiently large constant (e.g.20). There exists a Lipschitz mapping $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n^{\prime}}$ such that for all $\mathbf{u},\mathbf{v}\in Q$*

[TABLE]

A number of proofs and applications of this theorem have been proposed which in fact do not prove the theorem as such but rather create a probabilistic version of it, like e.g. [11, 1, 3, 12, 14, 10]. For an overview of Johnson-Lindenstrauss Lemma variants see e.g. [15].

Essentially the idea behind these probabilistic proofs is as follows: It is proven that the probability of reconstructing the length of a random vector from a projection onto a subspace within a reasonable error boundaries is high.

One then inverts the thinking and states that the probability of reconstructing the length of a given vector from a projection onto a (uniformly selected) random subspace within a reasonable error boundaries is high.

But uniform sampling of high dimensional subspaces is a hard task. So instead $n^{\prime}$ vectors with random coordinates are sampled from the original $n$-dimensional space and one uses them as a coordinate system in the $n^{\prime}$-dimensional subspace which is a much simpler process. One hopes that the sampled vectors will be orthogonal (and hence the coordinate system will be orthogonal) which in case of vectors with thousands of coordinates is reasonable. That means we create a matrix $M$ of $n^{\prime}$ rows and $n$ columns as follows: for each row $i$ we sample $n$ numbers from $\mathcal{N}(0,1)$ forming a row vector $\mathbf{a}^{T}_{i}$. We normalize it obtaining the row vector $\mathbf{b}^{T}_{i}=\mathbf{a}^{T}_{i}\cdot(\mathbf{a}^{T}_{i}\mathbf{a}_{i})^{-1/2}$. This becomes the $i$th row of the matrix $M$. Then for any data point $\mathbf{x}$ in the original space its random projection is obtained as $\mathbf{x^{\prime}}=M\mathbf{x}$.

Then the mapping we seek is the projection multiplied by a suitable factor.

It is claimed afterwards that this mapping is distance-preserving not only for a single vector, but also for large sets of points with some, usually very small probability, as Dasgupta and Gupta [11] maintain. Via applying the above process many times one can finally get the mapping $f$ that is needed. That is each time we sample a subspace from the space of subspaces and check if condition expressed by equation (1) holds for all the points, and if not, we sample again, while we have the reasonable hope that we will get the subspace of interest after a finite number of steps with probability that we assume.

In this paper we explore the following flaw of the mentioned approach: If we want to apply for example a $k$-means clustering algorithm, we are in fact not interested in resampling the subspaces in order to find a convenient one so that the distances are sufficiently preserved. Computation over and over again of $m^{2}/2$ distances between the points in the projected space may turn out to be much more expensive than computing $O(mk)$ distances during $k$-means clustering (if $m\gg k$) in the original space. In fact we are primarily interested in clustering data. But we do not have any criterion for the $k$-means algorithm that would say that this particular subspace is the right one via e.g. minimization of $k$-means criterion (and in fact for any other clustering algorithm).

Therefore, we rather seek a scheme that will allow us to say that by a certain random sampling we have already found the subspace that we sought with a sufficiently high probability. As far as we know, this is the first time such a problem has been posed.

To formulate claims concerning $k$-means, we need to introduce additional notation. Let us denote with $\mathfrak{C}$ a partition of $\mathfrak{Q}$ into $k$ clusters $\{C_{1},\dots,C_{k}\}$. For any $i\in\mathfrak{Q}$ let $\mathfrak{C}(i)$ denote the cluster $C_{j}$ to which $i$ belongs. For any set of objects $C_{j}$ let $\boldsymbol{\mu}(C_{j})=\frac{1}{|C_{j}|}\sum_{i\in C_{j}}\mathbf{x_{i}}$ and $\boldsymbol{\mu}^{\prime}(C_{j})=\frac{1}{|C_{j}|}\sum_{i\in C_{j}}\mathbf{x^{\prime}_{i}}$.

Under this notation the $k$-means cost function may be written as

[TABLE]

for the sets $Q,Q^{\prime}$.

Our contribution is as follows:

• •

We formulate and prove a set version of JL Lemma - see Theorem 6.

• •

Based on it we demonstrate that a good solution to $k$-means problem in the projected space is also a good one in the original space

• see Theorem 2.
• •

We show that local $k$-means minima in the original and the projected spaces match under proper conditions

• see Theorems 3, 4.
• •

We demonstrate that a perfect $k$-means algorithm in the projected space is a constant factor approximation of the global optimum in the original space

• see Theorem 5
• •

We prove that the projection preserves several clusterability properties

• see Theorems 9, 7, 8. 10 and 11.

For $k$-means in particular we make the following claim:

Theorem 2**.**

Let $Q$ be a set of $m$ representatives of objects from $\mathfrak{Q}$ in an $n$-dimensional orthogonal coordinate system $\mathcal{C}_{n}$. Let $\delta\in(0,\frac{1}{2})$, $\epsilon\in(0,1)$. and let

[TABLE]

Let $\mathcal{C}_{n^{\prime}}$ be a randomly selected (via sampling from a normal distribution) $n^{\prime}$-dimensional orthogonal coordinate system. Let the set $Q^{\prime}$ consist of $m$ objects such that for each $i\in\mathfrak{Q}$, $\mathbf{x^{\prime}}_{i}\in Q^{\prime}$ is a projection of $\mathbf{x}_{i}\in Q$ onto $\mathcal{C}_{n^{\prime}}$. If $\mathfrak{C}$ is a partition of $\mathfrak{Q}$, then

[TABLE]

*holds with probability of at least $1-\epsilon$.

Note that the inequality (5) can be rewitten as

[TABLE]

Furthermore

Theorem 3**.**

Under the assumptions and notation of Theorem 2, if the partition $\mathfrak{C}^{*}$ constitutes a local minimum of $\mathfrak{J}(Q,\mathfrak{C})$ over $\mathfrak{C}$ (in the original space) and if for any two clusters $g=2(1-\alpha)$ times half of the distance between their centres is the gap between these clusters, where $\alpha\in[0,1)$, and

[TABLE]

*($p$ to be defined later by inequality (14)) then this same partition is (in the projected space) also a local minimum of $\mathfrak{J}(Q^{\prime},\mathfrak{C})$ over $\mathfrak{C}$, with probability of at least $1-\epsilon$.

Theorem 4**.**

Under the assumptions and notation of Theorem 2, if the clustering $\mathfrak{C^{\prime}}^{*}$ constitutes a local minimum of $\mathfrak{J}(Q^{\prime},\mathfrak{C})$ over $\mathfrak{C}$ (in the projected space) and if for any two clusters $1-\alpha$ times the distance between their centres is the gap between these clusters, where $\alpha\in[0,1)$, and

[TABLE]

*then the very same partition $\mathfrak{C^{\prime}}^{*}$ is also (in the original space) a local minimum of $\mathfrak{J}(Q,\mathfrak{C})$ over $\mathfrak{C}$, with probability of at least $1-\epsilon$.

Theorem 5**.**

Under the assumptions and notation of Theorem 2, if $\mathfrak{C_{G}}$ denotes the clustering reaching the global optimum in the original space, and $\mathfrak{C^{\prime}_{G}}$ denotes the clustering reaching the global optimum in the projected space, then

[TABLE]

with probability of at least $1-\epsilon$.

That is the perfect $k$-means algorithm in the projected space is a constant factor approximation of $k$-means optimum in the original space.

We postpone the proof of the theorems 2-5 till section 3, as we need first to derive the basic theorem 6 in section 2 which is essentially based on the results reported by Dasgupta and Gupta [11].

Let us however stress at this point the significance of these theorems. Earlier forms of JL lemma required sampling of the coordinates over and over again222 Though in passing a similar result is claimed in Lemma 5.3 http://math.mit.edu/~bandeira/2015_18.S096_5_Johnson_Lindenstrauss.pdf, though without an explicit proof. , with quite a low success rate until a mapping is found fitting the error constraints. In our theorems, we need only one sampling in order to achieve the required success probability of selecting a suitable subspace to perform $k$-means. In Section 5 we illustrate this advantage by some numerical simulation results, showing at the same time the impact of various parameters of Jonson-Lindenstrauss Lemma on the dimensionality of the projected space. In Section 6 we recall the corresponding results of other authors.

In Section 4 we demonstrate an additional advantage of our version of JL lemma consisting in preservation of various clusterability criteria.

Section 7 contains some concluding remarks.

2 Derivation of the Set-Friendly

Johnson-Lindenstrauss Lemma

Let us present the process of seeking the mapping $f$ from Theorem 1 in a more detailed manner, so that we can then switch to our target of selecting the size of the subspace guaranteeing that the projected distances preserve their proportionality in the required range.

Let us consider first a single vector $\mathbf{x}=(x_{1},.x_{2},...,x_{n})$ of $n$ independent random variables drawn from the normal distribution $\mathcal{N}(0,1)$ with mean 0 and variance 1. Let $\mathbf{x^{\prime}}=(x_{1},.x_{2},...,x_{n^{\prime}})$, where $n^{\prime}<n$, be its projection onto the first $n^{\prime}$ coordinates.

Dasgupta and Gupta [11] in their Lemma 2.2 demonstrated that for a positive $\beta$

• •

if $\beta<1$ then

[TABLE]

• •

if $\beta>1$ then

[TABLE]

Now imagine we want to keep the error of squared length of $\mathbf{x}$ bounded within a range of $\pm\delta$ (relative error) upon projection, where $\delta\in(0,1)$. Then we get the probability

[TABLE]

This implies

[TABLE]

The same holds if we scale the vector $\mathbf{x}$.

Now if we have a sample consisting of $m$ points in space, without however a guarantee that coordinates are independent between the vectors then we want that the probability that squared distances between all of them lie within the relative range $\pm\delta$ is higher than

[TABLE]

for some failure probability333 We speak about a success if all the projected data points lie within the range defined by formula (1). Otherwise we speak about failure (even if only one data point lies outside this range). term $\epsilon\in(0,1)$.

To achieve this, it is sufficient that the following holds:

[TABLE]

Taking logarithm

[TABLE]

[TABLE]

We know444Please recall at this point the Taylor expansion $\ln(1+x)=x-x^{2}/2+x^{3}/3-x^{5}/5\dots$ which converges in the range (-1,1) and hence implies $\ln(1+x)<x$ for $x\in(-1,0)\cup(0,1)$ as we will refer to it discussing difference to JL theorems of other authors.

that $\ln(1+x)<x$ for $x>-1$ and $x\neq 0$, hence the above holds if

[TABLE]

[TABLE]

Recall that also we have $\ln(1-x)+x<0$ for $x<1$ and $x\neq 0$, threfore

[TABLE]

So finally, realizing that ${-\ln(1-\delta)-\delta}\geq{-\ln(1+\delta)+\delta}>0$, and that $\ln(m(m-1))<2\ln(m)$ we get as sufficient condition555 We substituted the denominator with a smaller positive number and the nominator with a larger positive number so that the fraction value increases so that a higher $n^{\prime}$ will be required than actually needed.

[TABLE]

Note that this expression does not depend on $n$ that is the number of dimensions in the projection is chosen independently of the original number of dimensions666 Though in passing a similar result is claimed in Lemma 5.3 http://math.mit.edu/~bandeira/2015_18.S096_5_Johnson_Lindenstrauss.pdf, though without an explicit proof. They propose that

$n^{\prime}\geq(2+r)\frac{2\ln(m)}{{-\ln(1+\delta)+\delta}}$

in order to get a failure rate below $m^{-r}$. In fact when we substitute $\epsilon=m^{-r}$, both formulas are the same. However, usage of $\epsilon$ alows for control of failure rate in the other theorems in this paper, while $r$ does not make this possibility obvious. Also fixing $r$ versus fixing $\epsilon$ impacts disadvantageously the growth rate of $n^{\prime}$ with $m$. .

So we are ready to formulate our major finding of this paper

Theorem 6**.**

Let $\delta\in(0,\frac{1}{2})$, $\epsilon\in(0,1)$. Let $Q\subset\mathbb{R}^{n}$ be a set of $m$ points in an $n$-dimensional orthogonal coordinate system $\mathcal{C}_{n}$ and let (as in formula (4))

[TABLE]

Let $\mathcal{C}_{n^{\prime}}$ be a randomly selected (via sampling from a normal distribution) $n^{\prime}$-dimensional orthogonal coordinate system. For each $\mathbf{v}\in Q$ let $\mathbf{v^{\prime}}$ be its projection onto $\mathcal{C}_{n^{\prime}}$. Then for all pairs $\mathbf{u},\mathbf{v}\in Q$

[TABLE]

holds with probability of at least $1-\epsilon$

3 Proofs of theorems 2-5

The permissible error $\delta$ will surely depend on the target application. Let us consider the context of $k$-means. First we claim for $k$-means, that the JL Lemma applies not only to data points but also to cluster centres.

Lemma 1**.**

Let $\delta\in(0,\frac{1}{2})$, $\epsilon\in(0,1)$. Let $Q\subset\mathbb{R}^{n}$ be a set of $m$ representatives of elements of $\mathfrak{Q}$ in an $n$-dimensional orthogonal coordinate system $\mathcal{C}_{n}$ and let the inequality (4) hold. Let $\mathcal{C}_{n^{\prime}}$ be a randomly selected (via sampling from a normal distribution) $n^{\prime}$-dimensional orthogonal coordinate system. For each $\mathbf{x}_{i}\in Q$ let $\mathbf{x_{i}^{\prime}}\in Q^{\prime}$ be its projection onto $\mathcal{C}_{n^{\prime}}$. Let $\mathfrak{C}$ be a partition of $\mathfrak{Q}$. Then for all data points $\mathbf{x_{i}}\in Q$

[TABLE]

hold with probability of at least $1-\epsilon$,

Proof.

As we know, data points under $k$-means are assigned to clusters having the closest cluster centre. On the other hand the cluster centre $\boldsymbol{\mu}$ is the average of all the data point representatives in the cluster.

Hence the cluster element $i$ has the squared distance to its cluster centre $\boldsymbol{\mu}(\mathfrak{C}(i))$ amounting to

[TABLE]

But according to Theorem 6

[TABLE]

Hence

[TABLE]

Note that here $\boldsymbol{\mu}^{\prime}(\mathfrak{C}(i))$ is not the projective image of $\boldsymbol{\mu}(\mathfrak{C}(i))$, but rather the centre of projected images of cluster elements.

∎

The Lemma 1 permits us to prove Theorem 2

Proof.

(Theorem 2) According to formula (13):

[TABLE]

Hence

[TABLE]

[TABLE]

Based on defining equations (2) and (3) we get the formula (5)

[TABLE]

∎

Let us now investigate the distance between centres of two clusters, say $C_{1},C_{2}$. Let their cardinalities amount to $m_{1},m_{2}$ respectively. Denote $C_{12}=C_{1}\cup C_{2}$. Consequently $m_{12}=|C_{12}|=m_{1}+m_{2}$. For a set $C_{j}$ let $VAR(C_{j})=\frac{1}{|C_{j}|}\sum_{i\in C_{j}}\|\mathbf{x_{i}}-\boldsymbol{\mu}(C_{j})\|^{2}$ and $VAR^{\prime}(C_{j})=\frac{1}{|C_{j}|}\sum_{i\in C_{j}}\|\mathbf{x^{\prime}_{i}}-\boldsymbol{\mu}^{\prime}(C_{j})\|^{2}$.

Therefore

[TABLE]

[TABLE]

By inserting a zero

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Via the same reasonig we get:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

As Apparently $\boldsymbol{\mu}(C_{12})=\frac{1}{|C_{12}|}\sum_{i\in C_{12}}\mathbf{x_{i}}$ $=\frac{1}{|C_{12}|}\left((\sum_{i\in C_{1}}\mathbf{x_{i}})+(\sum_{i\in C_{2}}\mathbf{x_{i}})\right)$ $=\frac{1}{|C_{12}|}\left(|C_{1}|\boldsymbol{\mu}(C_{1})+|C_{1}|\boldsymbol{\mu}(C_{1})\right)$ that is $\boldsymbol{\mu}(C_{12})=\frac{|C_{1}|}{|C_{12}|}\boldsymbol{\mu}(C_{1})+\frac{|C_{2}|}{|C_{12}|}\boldsymbol{\mu}(C_{2}$, we get

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

hence

[TABLE]

This leads immediately to

[TABLE]

which implies

[TABLE]

According to Lemma 1, applied to the set $C_{12}$ as a cluster,

[TABLE]

[TABLE]

[TABLE]

and with respect to $C_{1},C_{2}$ combined

[TABLE]

[TABLE]

[TABLE]

These two last equations mean that

[TABLE]

[TABLE]

[TABLE]

Let us assume that the quotient

[TABLE]

where $p$ is some positive number. So we have in effect

[TABLE]

Under balanced ball-shaped clusters $p$ does not exceed 1. So we have shown the lemma

Lemma 2**.**

Under the assumptions of preceding lemmas for any two clusters $C_{1},C_{2}$

[TABLE]

where $p$ depends on degree of balance between clusters and cluster shape, holds with probability at least $1-\epsilon$.

Now let us consider the choice of $\delta$ in such a way that with high probability no data point will be classified into some other cluster. We claim the following

Lemma 3**.**

*Consider two clusters $C_{1},C_{2}$. Let $\delta\in(0,\frac{1}{2})$, $\epsilon\in(0,1)$. Let $Q\subset\mathbb{R}^{n}$ be a set of $m$ points in an $n$-dimensional orthogonal coordinate system $\mathcal{C}_{n}$ and let the inequality (4) hold. Let $\mathcal{C}_{n^{\prime}}$ be a randomly selected (via sampling from a normal distribution) $n^{\prime}$-dimensional orthogonal coordinate system. For each $\mathbf{x}_{i}\in Q$ let $\mathbf{v^{\prime}}$ be its projection onto $\mathcal{C}_{n^{\prime}}$. For two clusters $C_{1},C_{2}$, obtained via $k$-means, in the original space let $\boldsymbol{\mu}_{1},\boldsymbol{\mu}_{2}$ be their centres and $\boldsymbol{\mu}_{1}^{\prime},\boldsymbol{\mu}_{2}^{\prime}$ be centres to the correspondings sets of projected cluster members. Furthermore let $d$ be the distance of the first cluster centre to the common border of both clusters and let the closest point of the first cluster to this border be at the distance of $\alpha d$ from its cluster centre as projected on the line connecting both cluster centres, where $\alpha\in(0,1)$.

Then all projected points of the first cluster are (each) closer to the centre of the set of projected points of the first than to the centre of the set of projected points of the second if*

[TABLE]

where $g=2(1-\alpha)$, with probability of at least $1-\epsilon$.

Proof.

Consider a data point $\mathbf{x}$ ”close” to the border between the two neighbouring clusters, on the line connecting the cluster centres, belonging to the first cluster, at a distance $\alpha d$ from its cluster centre, where $d$ is the distance of the first cluster centre to the border and $\alpha\in(0,1)$. The squared distance between cluster centres, under projection, can be ”reduced” by the factor $1-\delta$, (beside the factor $\frac{n}{n^{\prime}}$ which is common to all the points) whereas the squared distance of $\mathbf{x}$ to its cluster centre may be ”increased” by the factor $1+\delta$. This implies a relationship between the factor $\alpha$ and the error $\delta$.

If $\mathbf{x^{\prime}}$ should not cross the border between the clusters, the following needs to hold:

[TABLE]

which implies:

[TABLE]

As (see Lemma 1)

[TABLE]

and (see Lemma 2)

[TABLE]

we know that, for inequality (17) to hold, it is sufficient that:

[TABLE]

that is

[TABLE]

But $2(1-\alpha)d$ or $2(1-\alpha)$ can be viewed as absolute or relative gap between clusters. So if we expect a relative gap $g=2(1-\alpha)$ between clusters, we have to choose $\delta$ in such a way that

[TABLE]

Therefore

[TABLE]

∎

So we see that the decision on the permitted error depends on the size of the gap between clusters that we hope to observe.

The Lemma 3 allows us to prove Theorem 3 in a straight forward manner.

Proof.

(Theorem 3) Observe that in this theorem we impose the condition of this lemma on each cluster. So all projected points are closer to their set centres than to any other centre. So the $k$-means algorithm would get stuck at this clustering and hence we get at a local minimum. ∎

Lemma 4**.**

*Let $\delta\in(0,\frac{1}{2})$, $\epsilon\in(0,1)$. Let $Q\subset\mathbb{R}^{n}$ be a set of $m$ points in an $n$-dimensional orthogonal coordinate system $\mathcal{C}_{n}$ and let the inequility (4) hold. Let $\mathcal{C}_{n^{\prime}}$ be a randomly selected (via sampling from a normal distribution) $n^{\prime}$-dimensional orthogonal coordinate system. For each $\mathbf{x}_{i}\in Q$ let $\mathbf{v^{\prime}}$ be its projection onto $\mathcal{C}_{n^{\prime}}$. For any two $k$-means clusters $C_{1},C_{2}$ in the projected space let $\boldsymbol{\mu}_{1}^{\prime},\boldsymbol{\mu}_{2}^{\prime}$ be their centres in the projected space and $\boldsymbol{\mu}_{1},\boldsymbol{\mu}_{2}$ be centres to the corresponding sets of cluster members in the original space. Furthermore let $d$ be the distance of the first cluster centre to the common border of both clusters in the projected space and let the closest point of the first cluster to this border in that space be at the distances of $\alpha d$ from its cluster centre, where $\alpha\in[0,1)$.

Then all points of the first cluster in the original space are (each) closer to the centre of the set of points of the first than to the centre of the set of points of the second cluster in the original space if*

[TABLE]

with probability of at least $1-\epsilon$.

Proof.

Consider a data point $\mathbf{x^{\prime}}$ ”close” to the border between the two neighbouring clusters in the projected space, on the line connecting the cluster centres, belonging to the first cluster, at a distance $\alpha d$ from its cluster centre, where $d$ is the distance of the first cluster centre to the border and $\alpha\in(0,1)$. The squared distance between cluster centres, in original space, can be ”reduced” by the factor $(1+\delta)^{-1}$ (beside the factor $\frac{n^{\prime}}{n}$ which is common to all the points), whereas the squared distance of $\mathbf{x}$ to its cluster centre may be ”increased” by the factor $(1-\delta)^{-1}$ . This implies a relationship between the factor $\alpha$ and the error $\delta$.

If $\mathbf{x}$ (in the original space) should not cross the border between the clusters, the following needs to hold:

[TABLE]

which implies:

[TABLE]

As (see Lemma 1)

[TABLE]

and (see Lemma 2)

[TABLE]

Thus, we know that, for inequality (20) to hold, it is sufficient that:

[TABLE]

that is

[TABLE]

But $2(1-\alpha)d$ or $2(1-\alpha)$ can be viewed as absolute or relative gap between clusters. So if we want to have a relative gap $g=2(1-\alpha)$ between clusters, we have to choose $\delta$ in such a way that

[TABLE]

Therefore

[TABLE]

∎

The Lemma 4 allows us to prove Theorem 4 in a straight forward manner.

Proof.

(Theorem 4) Observe that in this theorem we impose the condition of this lemma on each cluster. So all original space points are closer to their set centres than to any other centre. So the $k$-means algorithm would get stuck at this clustering and hence we get at a local minimum. ∎

Having these results, we can go over to the proof of the Theorem 5.

Proof.

(Theorem 5) Let $\mathfrak{C_{G}}$ denote the clustering reaching the global optimum in the original space. Let $\mathfrak{C^{\prime}_{G}}$ denote the clustering reaching the global optimum in the projected space. From the Theorem 2 we have that

[TABLE]

On the other hand

[TABLE]

As $\mathfrak{C^{\prime}_{G}}$ is the global minimum in the projected space, hence

[TABLE]

So

[TABLE]

So

[TABLE]

Note that analogously, $\mathfrak{C_{G}}$ is the global minimum in the original space, hence

[TABLE]

[TABLE]

∎

4 Clusterability and the dimensionality reduction

In the literature a number of notions of so-called clusterability have been introduced. Under these notions of clusterability algorithms have been developed clustering the data nearly optimally in polynomial times, when some constraints are matched by the clusterability parameters.

It seems therefore worth to have a look at the issue if the aforementioned projection technique would affect the clusterability property of the data sets.

Let us consider, as representatives, the following notions of clusterability, present in the literature:

• •

Perturbation Robustness meaning that small perturbations of distances / positions in space of set elements do not result in a change of the optimal clustering for that data set. Two brands may be distinguished: additive [2] and multiplicative ones [9] (the limit of perturbation is upper-bounded either by an absolute value or by a coefficient).

The $s$-Multiplicative Perturbation Robustness ($0<s<1)$ holds for a data set with $d_{1}$ being its distance function if the following holds. Let $\mathfrak{C}$ be an optimal clustering of data points for this distance. Let $d_{2}$ be any distance function over the same set of points such that for any two points $\mathbf{u},\mathbf{v}$, $s\cdot d_{1}(\mathbf{u},\mathbf{v})<d_{2}(\mathbf{u},\mathbf{v})<\frac{1}{s}\cdot d_{1}(\mathbf{u},\mathbf{v})$. Then the same clustering $\mathfrak{C}$ is optimal under the distance function $d_{2}$.

The $s$-Additive Perturbation Robustness ($0<s<1)$ holds for a data set with $d_{1}$ being its distance function if the following holds. Let $\mathfrak{C}$ be an optimal clustering of data points for this distance. Let $d_{2}$ be any distance function over the same set of points such that for any two points $\mathbf{u},\mathbf{v}$, $d_{1}(\mathbf{u},\mathbf{v})-s<d_{2}(\mathbf{u},\mathbf{v})<\cdot d_{1}(\mathbf{u},\mathbf{v})+s$. Then the same clustering $\mathfrak{C}$ is optimal under the distance function $d_{2}$.

Subsequently we are interested only in the multiplicative version.

• •

$\sigma$-Separatedness [16] meaning that the cost $\mathfrak{J}(Q,\mathfrak{C}_{k})$ of optimal clustering $\mathfrak{C}_{k}$ of the data set $Q$ into $k$ clusters is less than $\sigma^{2}$ ($0<\sigma<1$) times the cost $\mathfrak{J}(Q,\mathfrak{C}_{k-1})$ of optimal clustering $\mathfrak{C}_{k-1}$ into $k-1$ clusters $\mathfrak{J}(Q,\mathfrak{C}_{k-1})<\sigma^{2}\mathfrak{J}(Q,\mathfrak{C}_{k-1})$

• •

$(c,\sigma)$-Approximation-Stability* [7] meaning that if the cost function values of two partitions $\mathfrak{C}_{a},\mathfrak{C}_{b}$ differ by at most the factor $c>1$ (that is $c\cdot\mathfrak{J}(Q,\mathfrak{C}_{a})\geq\mathfrak{J}(Q,\mathfrak{C}_{b})$ and $c\cdot\mathfrak{J}(Q,\mathfrak{C}_{b})\geq\mathfrak{J}(Q,\mathfrak{C}_{a})$), then the distance (in some space) between the partitions is at most $\sigma$ ($d(\mathfrak{C}_{a},\mathfrak{C}_{b})<\sigma$ for some distance function $d$ between partiitions). As Ben-David [8] recalls, this implies the uniqueness of optimal solution.

• •

$\beta$-Centre Stability* [6] meaning, for any centric clustering, that the distance of an element to its cluster centre is $\beta>1$ times smaller than the distance to any other cluster centre under optimal clustering.

• •

$(1+\beta)$ Weak Deletion Stability [5] ($\beta>0$) meaning that given an optimal cost function value $OPT$ for $k$ centric clusters, the cost function of a clustering obtained by deleting one of the cluster centres and assigning elements of that cluster to one of the remaining clusters should be bigger than $(1+\beta)\cdot OPT$.

Let us first have a look at the $\sigma$-Separatedness. Let $\mathfrak{C_{G,k}}$ denote an optimal clustering into $k$ clusters in the original space and $\mathfrak{C^{\prime}_{G,k}}$ in the projected space. From properties of $k$-means we know that $\mathfrak{J}(Q,\mathfrak{C_{G,k}})\leq\mathfrak{J}(Q,\mathfrak{C_{G,k-1}})$ and $\mathfrak{J}(Q^{\prime},\mathfrak{C^{\prime}_{G,k}})\leq\mathfrak{J}(Q^{\prime},\mathfrak{C^{\prime}_{G,k-1}})$. From theorem 5 we know that

[TABLE]

and

[TABLE]

$\sigma$-Separatedness implies that

[TABLE]

[TABLE]

This implies

[TABLE]

So we claim

Theorem 7**.**

Under the assumptions and notation of Theorem 2, If the data set $Q$ has the property of $\sigma$-Separatedness in the original space, then with probability at least $1-\epsilon$ it has the property of $\sigma\sqrt{\frac{1+\delta}{1-\delta}}$-Separatedness in the projected space.

The fact that this Separatedness increases is of course a defficiency, because clustring algorithms require as low Separatedness as possible (because the clusters are then better separated).

Let us turn to the $(c,\sigma)$-Approximation-Stability. We can reformulate it as follows: if the distance (in some space) between the partitions is more than $\sigma$ then the cost function values of two partitions differ by at least the factor $c>1$. Consider now two partitions $C_{1},C_{2}$, with distance over $\sigma$ in some abstract partition space, not related to the embedding spaces. Then in the original space the following must hold.

[TABLE]

Under the projection we get

[TABLE]

[TABLE]

This result means that

Theorem 8**.**

Under the assumptions and notation of Theorem 2, if the data set $Q$ has the property of $(c,\sigma)$-Approximation- Stability in the original space, then with probability at least $1-\epsilon$ it has the property of $(c\cdot\frac{1-\delta}{1+\delta},\sigma$-Approximation Stability property in the projected space.

Let us now consider $s$-Multiplicative Perturbation Stability. We claim that

Lemma 5**.**

If the data set $Q$ has the property of $\sqrt{s}$-Multiplicative Perturbation Robustness under the distance $\sqrt{d_{1}}$, and the set $Q_{p}$ is its perturbation with distance $\sqrt{d_{2}}$ such that $\nu d_{1}\leq d_{2}\leq\frac{1}{\nu}d_{1}$, and $s=\nu\cdot s_{p}$, where $0<\nu,s_{p}<1$, then set $Q_{p}$ has the property of $\sqrt{s_{p}}$-Multiplicative Perturbation Robustness

Proof.

Apparently $Q_{p}$ is a perturbation of $Q$ such that both share same optimal clustering. Let $Q_{q}$ be a perturbation of $Q_{p}$, with distance $\sqrt{d_{3}}$, such that $s_{p}d_{2}\leq d_{3}\leq\frac{1}{s_{p}}d_{2}$. Then $sd_{1}=s_{p}\nu d_{1}\leq s_{p}d_{2}\leq d_{3}\leq\frac{1}{s_{p}}d_{2}\leq\frac{1}{s_{p}\nu}d_{1}=\frac{1}{s}d_{1}$ that is $Q_{q}$ is a perturbation of $Q$ such that both share same optimal clustering. So $Q_{p}$ and $Q_{q}$ share common optimal clustering, hence $Q_{p}$ has the property of $\sqrt{s_{p}}$-Multiplicative Perturbation Robustness ∎

We claim that

Lemma 6**.**

Under the assumptions and notation of Theorem 2, if the data set $Q$ has the property of $\sqrt{s}$-Multiplicative Perturbation Robustness with $s<1-\delta$, and if $\mathfrak{C_{G}}$ is the global optimum of $k$-means in $Q$, then it is also the global optimum in $Q^{\prime}$ with probability at least $1-\epsilon$

Proof.

Assume the contrary that is that in $Q^{\prime}$ some other clustering $\mathfrak{C^{\prime}_{G}}$ is the global optimum. Let us define the distance $\sqrt{d_{1}(i,j)}=\|\mathbf{x_{i}}-\mathbf{x_{j}}\|$ and $\sqrt{d_{2}(i,j)}=\frac{n}{n^{\prime}}\|\mathbf{x_{i}^{\prime}}-\mathbf{x_{j}^{\prime}}\|$. The distance $\sqrt{d_{2}}$ is a realistic distance in the coordinate system $\mathcal{C}$ as we assume $n>n^{\prime}$. As the $k$-means optimum does not change under rescaling, so $\mathfrak{C^{\prime}_{G}}$ is also an optimal solution for clustering task under $d_{2}$. But

[TABLE]

hence the distance $\sqrt{d_{2}}$ is a perturbation of $\sqrt{d_{1}}$ and hence $\mathfrak{C_{G}}$ should be optimal under $\sqrt{d_{2}}$ also. We get a contradiction. So the claim of the lemma must be true. ∎

This implies that

Theorem 9**.**

Under the assumptions and notation of Theorem 2, if the data set $Q$ has the property of $\sqrt{s}$-Multiplicative Perturbation Robustness with factor $s<s_{p}\nu\frac{(1-\delta)^{2}}{1+\delta}$ ($0<s_{p},\nu<1$) in the original space, then with probability at least $1-2\epsilon$ it has the property of $\sqrt{s_{p}}$-Multiplicative Perturbation Robustness in the projected space.

Proof.

The Lemma 6 implies that the global optima of the original and projected spaces are identical. So assume that in the original space for the distance $\sqrt{d_{1}(i,j)}=\|\mathbf{x_{i}}-\mathbf{x_{j}}\|$ $\mathfrak{C_{G}}$ is the optimal clustering. Then under projection $\sqrt{d_{1}^{\prime}(i,j)}=\|\mathbf{x_{i}^{\prime}}-\mathbf{x_{j}^{\prime}}\|$ we have the same optimal clustering.

For a perturbation with factor $s_{p}$ in the projected space define the distance $\sqrt{d_{2}^{\prime}(i,j)}=\|\mathbf{y_{i}^{\prime}}-\mathbf{y_{j}^{\prime}}\|$ where for any $i$ let $\mathbf{y_{i}^{\prime}}$ be a perturbation of $\mathbf{x_{i}^{\prime}}$. We will be done if we can demonstrate that $\sqrt{d_{2}^{\prime}}$ yields the same optimum in the projected space as $\sqrt{d_{1}^{\prime}}$ does. For any $i$ let $\mathbf{y_{i}}$ be some point in the original space such that $\mathbf{y_{i}^{\prime}}$ is its projection to the projected space. We will treat $\mathbf{y_{i}}$ as an image of $\mathbf{x_{i}}$ and will subsequently show that the set of these points $\mathbf{y_{i}}$ can be treated as a perturbation of $\mathbf{x_{i}}$ with the factor $\sqrt{s}$.

For each counterpart $\sqrt{d_{2}(i,j)}=\|\mathbf{y_{i}}-\mathbf{y_{j}}\|$ of $\sqrt{d_{2}^{\prime}}$ in original space $(1+\delta)^{-1}\frac{n}{n^{\prime}}d_{2}^{\prime}(i,j)\leq d_{2}(i,j)\leq(1-\delta)^{-1}\frac{n}{n^{\prime}}d_{2}^{\prime}(i,j)$ holds. As $s_{p}d_{1}^{\prime}(i,j)\leq d_{2}^{\prime}(i,j)\leq(s_{p})^{-1}d_{1}^{\prime}(i,j)$ and $(1-\delta)d_{1}(i,j)\leq\frac{n}{n^{\prime}}d^{\prime}_{1}(i,j)\leq(1+\delta)d_{1}(i,j)$ we obtain

[TABLE]

[TABLE]

[TABLE]

So $\sqrt{d_{2}}$ is a perturbation of $\sqrt{d_{1}}$ with the factor $\sqrt{s}$. $\sqrt{d_{1}}$ is $\sqrt{s}$-multiplicative perturbation robust, therefore both have the same optimal solution $\mathfrak{C_{G}}$. Furthermore $\sqrt{d_{2}}$ has the property of $\sqrt{\nu(1-\delta)}$-Multiplicative Robustness (see Lemma 5). Therefore its counterpart $\sqrt{d_{2}^{\prime}}$ has the same optimum clustering $\mathfrak{C_{G}}$ as $\sqrt{d_{2}}$ (see Lemma 6), hence as $\sqrt{d_{1}}$, hence as $\sqrt{d_{1}^{\prime}}$.

Recall that $\sqrt{d_{2}^{\prime}}$ was selected as any perturbation of $\sqrt{d_{1}^{\prime}}$ with factor $\sqrt{s_{p}}$. And it turned out that it yields the same optimal solution as $\sqrt{d_{1}^{\prime}}$. So with high probability (factor 2 is taken as we deal with two data sets, comprising points $\mathbf{x}_{i}$ and $\mathbf{y}_{i}$) $\sqrt{d_{1}^{\prime}}$ possesses $\sqrt{s_{p}}$-Multiplicative Perturbation Robustness in the projected space. ∎

We claim

Theorem 10**.**

Under the assumptions and notation of Theorem 2, if the data set $Q$ has both the property of $\beta$-Centre Stability and $\sqrt{s}$-Multiplicative Perturbation Robustness with $s<1-\delta$ in the original space, then with probability at least $1-\epsilon$ it has the property of $\beta\sqrt{\frac{1-\delta}{1+\delta}}$-Centre Stability in the projected space.

Proof.

The $\sqrt{s}$-Multiplicative Perturbation Robustness ensures that both the original and the projected space share same optimal clustering $\mathfrak{C}$.

Consider a data point $\mathbf{x_{i}}$ and a cluster $C\in\mathfrak{C}$ not containing $i$. Then $\mathbf{x_{i}}$, $\boldsymbol{\mu}(C)$ and $\boldsymbol{\mu}(C\cup\{i\})$ are colinear. So are $\mathbf{x_{i}^{\prime}}$, $\boldsymbol{\mu}^{\prime}(C)$ and $\boldsymbol{\mu}^{\prime}(C\cup\{i\})$, that is the respective (linear) projections. Furthermore $\frac{\|\mathbf{x_{i}}-\boldsymbol{\mu}(C\cup\{i\})\|}{\|\boldsymbol{\mu}(C)-\boldsymbol{\mu}(C\cup\{i\})\|}=\frac{|C|}{1}$, hence $\|\mathbf{x_{i}}-\boldsymbol{\mu}(C)\|=\frac{|C|+1}{|C|}\|\mathbf{x_{i}}-\boldsymbol{\mu}(C\cup\{i\})\|$. Likewise $\frac{\|\mathbf{x_{i}^{\prime}}-\boldsymbol{\mu}^{\prime}(C\cup\{i\})\|}{\|\boldsymbol{\mu}^{\prime}(C)-\boldsymbol{\mu}^{\prime}(C\cup\{i\})\|}=\frac{|C|}{1}$. Upon projection the distance to own cluster centre can increase relatively by $\sqrt{1+\delta}$ and to the $C\cup\{i\}$ centre can decrease by $\sqrt{1-\delta}$, see Lemma 1. That means $\|\mathbf{x_{i}^{\prime}}-\boldsymbol{\mu}^{\prime}(\mathfrak{C}(i))\|^{2}\leq(1+\delta)\frac{n^{\prime}}{n}\|\mathbf{x_{i}}-\boldsymbol{\mu}(\mathfrak{C}(i))\|^{2}$ and $(1-\delta)^{-1}\|\mathbf{x_{i}^{\prime}}-\boldsymbol{\mu}^{\prime}(C\cup\{i\})\|^{2}\geq\frac{n^{\prime}}{n}\|\mathbf{x_{i}}-\boldsymbol{\mu}(C\cup\{i\})\|^{2}$. Due to the aforementioned relations $\|\mathbf{x_{i}^{\prime}}-\boldsymbol{\mu}^{\prime}(C)\|^{2}\geq(1-\delta)\frac{n^{\prime}}{n}\|\mathbf{x_{i}}-\boldsymbol{\mu}(C)\|^{2}$. Due to $\beta$-Centre-Stability in the original space we had: $\beta^{2}\|\mathbf{x_{i}}-\boldsymbol{\mu}(\mathfrak{C}(i))\|^{2}<\|\mathbf{x_{i}}-\boldsymbol{\mu}(C)\|^{2}$. Due to the aforementioned relations we have

[TABLE]

[TABLE]

That is $\|\mathbf{x_{i}^{\prime}}-\boldsymbol{\mu}^{\prime}(C)\|>\beta\sqrt{\frac{1-\delta}{1+\delta}}\|\mathbf{x_{i}^{\prime}}-\boldsymbol{\mu}^{\prime}(\mathfrak{C}(i))\|$ Hence the data centre stability can drop to $\beta\sqrt{\frac{1-\delta}{1+\delta}}$. ∎

We claim

Theorem 11**.**

Under the assumptions and notation of Theorem 2, if the data set $Q$ has both the property of $(1+\beta)$ Weak Deletion Stability and $\sqrt{s}$-Multiplicative Perturbation Robustness with $s<1-\delta$ in the original space, then with probability at least $1-\epsilon$ it has the property of $(1+\beta)\frac{1-\delta}{1+\delta}$ Weak Deletion Stability in the projected space.

Proof.

The $\sqrt{s}$-Multiplicative Perturbation Robustness ensures that both original and the projected space share same optimal clustering. Let this optimal clustering be called $\mathfrak{C_{o}}$. By $\mathfrak{C}$ denote any clustering obtained from $\mathfrak{C_{o}}$ by deletion of one cluster centre and assigning cluster elements to one of the remaining clusters. By the assumption of (1+$\beta$)-Weak Deletion stability $(1+\beta)\mathfrak{J}(Q,\mathfrak{C_{o}})\leq\mathfrak{J}(Q,\mathfrak{C})$.

Theorem 2 implies that $(1-\delta)\frac{n^{\prime}}{n}\mathfrak{J}(Q,\mathfrak{C})\leq\mathfrak{J}(Q^{\prime},\mathfrak{C})$ and $(1+\delta)^{-1}\mathfrak{J}(Q^{\prime},\mathfrak{C_{o}})\leq\frac{n^{\prime}}{n}\mathfrak{J}(Q,\mathfrak{C_{o}})$.

Therefore $\mathfrak{J}(Q^{\prime},\mathfrak{C})\geq(1-\delta)\frac{n^{\prime}}{n}\mathfrak{J}(Q,\mathfrak{C})\geq(1+\beta)(1-\delta)\frac{n^{\prime}}{n}\mathfrak{J}(Q,\mathfrak{C_{o}})\geq(1+\beta)(1-\delta)(1+\delta)^{-1}\mathfrak{J}(Q^{\prime},\mathfrak{C_{o}})$ which implies the claim. ∎

5 Numerical Experiments on Some Aspects of Our Approach

Note that we have two formulas for computing the reduced space dimensionality $n^{\prime}$, the formula (11) and (4). The latter does not engage the original dimensionality $n$, while it is explicit in $n^{\prime}$. The value of $n^{\prime}$ in the former depends on $n$, however $n^{\prime}$ can be only computed iteratively.

Let us investigate the differences between $n^{\prime}$ computation in both cases. Let us check the impact of the following parameters: $n$ - the original dimensionality (see table 4 and figure 4), $\delta$ - the limitation of deviation of the distances between data points in the original and the reduced space (see table 3 and figure 3), $m$ - the sample size (see table 1 and figure 1), as well as $\epsilon$ - the maximum failure probability of the ”JL” transformation (see table 2 and figure 2). Note that in all figures the X-axis is on log scale.

As visible in figure 4 the value of $n^{\prime}$ from the explicit formula does not depend on the original dimensionality $n$. The value computed from the implicit formula approaches the explicit value quite quickly with the growing dimensionality $n$.

On the other hand, the implicit $n^{\prime}$ departs from the explicit one with growing sample size $m$, as visible in fig. 1. Both grow with increasing $m$.

In fig. 2 we see that when we increase the acceptable failure rate $\epsilon$, the requested dimensionality $n^{\prime}$ drops, whereby the implicit one approaches the explicit one.

Fig. 3 shows that the requested dimensionality drops quite quickly with increased relative error range $\delta$ till a kind of saturation is achieved. At extreme ends of $\delta$ implicit and explicit $n^{\prime}$ formulas converge to one another.

The behaviour of explicit $n^{\prime}$ is not surprising, as it is visible directly from the formula (4). The important insight here is however the required dimensionality of the projected data, of hundreds of thousands for realistic $\epsilon,\delta$. So the random projection via the Johnson-Lindenstrauss Lemma is not yet another dimensionality reduction technique. It is suitable for cases where techniques like PCA are not feasible computationally.

The behaviour of implicit $n^{\prime}$ for the case of increasing original dimensionality $n$ is as expected - the explicit $n^{\prime}$ reflects the ”in the limit” behaviour of the implicit formulation. The convergence for extreme values of $\delta$ is intriguing. The discrepancy for $\epsilon$ and the divergence for growing $m$ indicate that there is still space for better explicit formulas on $n^{\prime}$. Especially it is worth investigating for increasing $m$ as the processing becomes more expensive in the original space when $m$ is increasing.

In order to give an impression how effective the random projection is, see fig. 5. It illustrates the distribution of discrepancies between squared distances in the projected and in the original spaces. The discrepancies are expressed as

[TABLE]

One can see that they correspond quite well to the imposed constraints.

As the application for $k$-means clustering, we see in fig. 6 that the bigger the relative gap between clusters, the larger the error value $\delta$ is permitted, if class membership shall not be distorted by the projection.

6 Previous work

Note that if we would set $\epsilon$ (close) to 1, and expand by Taylor method the $\ln$ function in denominator of the inequality (4) to up to three terms then we get the value of $n^{\prime}$ from equation (2.1) from the paper [11]:

[TABLE]

Note, however, that setting $\epsilon$ to a value close to 1 does not make sense as we want to keep rare the event that the data does not fit the interval we are imposing.

Though one may be tempted to view our results as formally similar to those of Dasgupta and Gupta, there is one major difference. Let us first recall that the original proof of Johnson and Lindenstrauss [13] is probabilistic, showing that projecting the $m$ -point subset onto a random subspace of $O(\ln m/\epsilon^{2})$ dimensions only changes the (squared) distances between points by at most $1-\delta$ with positive probability. Dasgupta and Gupta showed that this probability is at least $1/m$, which is not much indeed. In order to get failure probability $\epsilon$ below say 0.05%, one needs to repeat the random projection and checking of distances $r$ times, with such $r$ that $\epsilon>(1-\frac{1}{m})^{r}$. In case of $m=1,000$ this means over $r=2,995$ repetitions, and with $m=1,000,000$ - over $r=2,995,000$ repetitions,

In this paper we have shown that this success probability can be raised to $1-\epsilon$ for an $\epsilon$ given in advance. Hereby the increase of target dimensionality is small enough compared to Dasgupta and Gupta formula, that our random projection method is orders of magnitude more efficient. A detailed comparison is contained in the tables 5, 6, 7, 8. We present in these tables $n^{\prime}$ computed using our formulas with those proposed by Dasgupta and Gupta as well as we present the required number of repetition of projection onto sampled subspaces in order to obtain a faithful distance discrepancies with reasonable probability. Dasgupta and Gupta generally obtain several times lower number of dimensions. However, as stated in the introduction, the number of repeated samplings annihilates this advantage and in fact a much higher burden when clustering is to be expected.

Note that the choice of $n^{\prime}$ has been estimated by [1]

[TABLE]

where $\gamma$ is some positive number. They propose a projection based on two or three discrete values randomly assigned instead of ones from normal distribution. With the quantity $\gamma$ they control the probability that a single element of the set $Q$ leaves the predefined interval $\pm\delta$. They do not bother about controlling the probability that none of the elements leaves the interval of interest. Rather, they derive expected values of various moments.

Larsen and Nelson [14] concentrate on finding the highest value of $n^{\prime}$ for which Johnson-Lindenstrauss Lemma does not hold demonstrating that the value they found is the tightest even for non-linear mappings $f$. Though not directly related to our research, they discuss the other side of the coin, that is the dimensionality below which at least one point of the data set has to violate the constraints.

7 Conclusions

In this paper we investigated a novel aspect of the well known and widely explored and exploited Johnson-Lindenstrauss lemma on the possibility of dimensionality reduction by projection onto a random subspace.

The original formulation means in practice that we have to check whether or not we have found a proper transformation $f$ leading to error bounds within required range for all pairs of points, and if necessary (and it is theoretically necessary very frequently), to repeat the random projection process over and over again.

We have shown here that it is possible to determine in advance the choice of dimensionality in the random projection process as to assure with desired certainty that none of the points of the data set violates restrictions on error bounds. This new formulation can be of importance for many data mining applications, like clustering, where the distortion of distances influences the results in a subtle way (e.g. $k$-means clustering).

Via some numerical examples we have pointed at the real application areas of this kind of projections, that is problems with high number of dimensions, starting with dozens of thousands and hundreds of thousands of dimensions.

Additionally, our reformulation of the JL Lemma permits to preserve some well-known clusterability properties at the projection.