The Informativeness of K -Means for Learning Mixture Models

TL;DR

This paper investigates the theoretical limits of k-means clustering for learning mixture models, showing conditions under which the algorithm's solutions closely approximate the true underlying data distribution, even with dimensionality reduction.

Contribution

It provides new information-theoretic conditions under which k-means yields clusterings close to the true mixture components, extending to log-concave distributions and reduced dimensions.

Findings

Optimal clusterings are close to true clusters under certain conditions.

Dimensionality reduction preserves the informativeness of k-means.

Results apply to spherical Gaussian and log-concave distributions.

Abstract

The learning of mixture models can be viewed as a clustering problem. Indeed, given data samples independently generated from a mixture of distributions, we often would like to find the {\it correct target clustering} of the samples according to which component distribution they were generated from. For a clustering problem, practitioners often choose to use the simple $k$-means algorithm. $k$-means attempts to find an {\it optimal clustering} that minimizes the sum-of-squares distance between each point and its cluster center. In this paper, we consider fundamental (i.e., information-theoretic) limits of the solutions (clusterings) obtained by optimizing the sum-of-squares distance. In particular, we provide sufficient conditions for the closeness of any optimal clustering and the correct target clustering assuming that the data samples are generated from a mixture of spherical Gaussian distributions. We also generalize our results to log-concave distributions. Moreover, we show that under similar or even weaker conditions on the mixture model, any optimal clustering for the samples with reduced dimensionality is also close to the correct target clustering. These results provide intuition for the informativeness of $k$-means (with and without dimensionality reduction) as an algorithm for learning mixture models.

Tables

Table 1. TABLE I: Summary of Feature Extraction Techniques

Technique	Reference	Dimensions	$\gamma$
PCA/SVD	Drineas et al. [42] Cohen et al. [25]	$K$ $\lceil K/ϵ\rceil$	2 $1+ϵ$
random projection	Cohen et al. [25]	$O\left(K/{ϵ}^2\right)$	$1+ϵ$
randomized SVD	Cohen et al. [25]	$\lceil K/ϵ\rceil$	$1+ϵ$

Equations

$$\mathcal{D}(\mathbf{V},\mathcal{C}):=\sum _{k=1}^K\sum _{n\in {\mathcal{C}}_k}\Vert {\mathbf{v}}_n-{\mathbf{c}}_k{\Vert }_2^2,$$

$$\mathcal{D}(\mathbf{V},{\mathcal{C}}^{\mathrm{opt}})=\underset{\mathcal{C}}{\min }\mathcal{D}(\mathbf{V},\mathcal{C}),$$

$$\Vert {\mathbf{u}}_i-{\mathbf{u}}_j{\Vert }_2$$

$$\mathcal{D}(\mathbf{V},\mathcal{C})=\Vert \mathbf{V}-\mathbf{V}{\bar{\mathbf{H}}}^T\bar{\mathbf{H}}{\Vert }_{\mathrm{F}}^2=\Vert \mathbf{V}{\Vert }_{\mathrm{F}}^2-\mathrm{tr}(\bar{\mathbf{H}}{\mathbf{V}}^T\mathbf{V}{\bar{\mathbf{H}}}^T).$$

$$\mathcal{D}(\mathbf{V},\mathcal{C})\ge {\mathcal{D}}^{\ast }(\mathbf{V}):=\mathrm{tr}(\mathbf{S})-\sum _{k=1}^{K-1}{\lambda }_k(\mathbf{S}),$$

$${\mathrm{d}}_{\mathrm{ME}}({\mathcal{C}}^{(1)},{\mathcal{C}}^{(2)}):=1-\frac{1}{N}\underset{\pi \in {\mathcal{P}}_K}{\max }\sum _{k=1}^K\left|{\mathcal{C}}_k^{(1)}\cap {\mathcal{C}}_{\pi (k)}^{(2)}\right|,$$

$$\tau (\delta ,{\delta }^{\prime })$$

$$\tau (\delta )$$

$$\delta :=\frac{\mathcal{D}(\mathbf{V},\mathcal{C})-{\mathcal{D}}^{\ast }(\mathbf{V})}{{\lambda }_{K-1}(\mathbf{S})-{\lambda }_K(\mathbf{S})},\text{mbox}and{\delta }^{\prime }:=\frac{\mathcal{D}(\mathbf{V},{\mathcal{C}}^{\prime })-{\mathcal{D}}^{\ast }(\mathbf{V})}{{\lambda }_{K-1}(\mathbf{S})-{\lambda }_K(\mathbf{S})}.$$

$${\mathrm{d}}_{\mathrm{ME}}(\mathcal{C},{\mathcal{C}}^{\prime })\le \tau (\delta ,{\delta }^{\prime })p_{\max }.$$

$${\mathrm{d}}_{\mathrm{ME}}(\mathcal{C},{\mathcal{C}}^{\mathrm{opt}})\le p_{\max }\tau (\delta ),$$

$$\mathcal{D}(\mathbf{V},{\tilde{\mathcal{C}}}^{\mathrm{opt}})\le \gamma \mathcal{D}(\mathbf{V},{\mathcal{C}}^{\mathrm{opt}}).$$

$$\mathbf{\Sigma }:=\sum _{k=1}^K{w}_k\left({\mathbf{u}}_k{\mathbf{u}}_k^T+{\mathbf{\Sigma }}_k\right),\text{mbox}and{\mathbf{\Sigma }}_0:=\sum _{k=1}^K{w}_k{\mathbf{u}}_k{\mathbf{u}}_k^T.$$

$$\bar{\mathbf{\Sigma }}$$

$$\zeta (p):=\frac{p}{1+\sqrt{1-\frac{2p}{K-1}}}.$$

$${\delta }_0:=\frac{(K-1){\bar{\sigma }}^2}{{\lambda }_{\min }}<\zeta (w_{\min }).$$

$$ϵ$$

$${\mathrm{d}}_{\mathrm{ME}}(\mathcal{C},{\mathcal{C}}^{\mathrm{opt}})\le \tau \left(\frac{(K-1){\bar{\sigma }}^2+ϵ}{{\lambda }_{\min }-ϵ}\right)(w_{\max }+ϵ),$$

$$\Vert {\mathbf{u}}_1-{\mathbf{u}}_2{\Vert }_2>\frac{\bar{\sigma }}{\sqrt{w_1{w}_2\zeta (w_{\min })}},$$

$$\frac{2K(K-1){\bar{\sigma }}^2}{\Vert {\mathbf{u}}_i-{\mathbf{u}}_j{\Vert }_2^2}<\frac{1}{2K},$$

$$\Vert {\mathbf{u}}_i-{\mathbf{u}}_j{\Vert }_2>2K\sqrt{K-1}\bar{\sigma }.$$

$${\delta }_0=\frac{(K-1{)}^2{\bar{\sigma }}^2}{d^2}.$$

$$d_{\mathrm{ME}}(\mathcal{C},{\mathcal{C}}^{\mathrm{opt}})\le \tau ({\delta }_0)w_{\max }\le \frac{2(K-1{)}^2{\bar{\sigma }}^2}{K{d}^2}.$$

$${\delta }_1:=\frac{(K-1){\bar{\sigma }}^2}{{\lambda }_{\min }+{\bar{\sigma }}^2}<\zeta (w_{\min }).$$

$$ϵ$$

$${\mathrm{d}}_{\mathrm{ME}}(\mathcal{C},{\tilde{\mathcal{C}}}^{\mathrm{opt}})\le \tau \left(\frac{(K-1){\bar{\sigma }}^2+ϵ}{{\lambda }_{\min }+{\bar{\sigma }}^2-ϵ}\right)(w_{\max }+ϵ),$$

$${\sigma }_k^2$$

$$|{\lambda }_m(\mathbf{A}+\mathbf{E})-{\lambda }_m(\mathbf{A})|\le \Vert \mathbf{E}{\Vert }_2$$

$$\mathbb{E}\left(\mathbf{x}{\mathbf{x}}^T\right)$$

$$\mathbb{E}\left((\mathbf{x}-\bar{\mathbf{u}})(\mathbf{x}-\bar{\mathbf{u}}{)}^T\right)=\sum _{k=1}^K{w}_k\left(({\mathbf{u}}_k-\bar{\mathbf{u}})({\mathbf{u}}_k-\bar{\mathbf{u}}{)}^T+{\mathbf{\Sigma }}_k\right)=\bar{\mathbf{\Sigma }}.$$

Full text

I Introduction

Suppose there are $K$ unknown distributions $F_{1},F_{2},\ldots,F_{K}$ and a probability vector $\mathbf{w}:=[w_{1},w_{2},\ldots,w_{K}]$. The corresponding $K$-component mixture model is a generative model that assumes data samples are independently sampled such that the probability that each sample is generated from the $k$-th component is $w_{k}$, the mixing weight for the $k$-th component. Suppose that $\mathbf{v}_{1},\mathbf{v}_{2},\ldots,\mathbf{v}_{N}$ are samples independently generated from a $K$-component mixture model, the correct target clustering $\mathscr{C}:=\{\mathscr{C}_{1},\mathscr{C}_{2},\ldots,\mathscr{C}_{K}\}$ satisfies the condition that $n\in\mathscr{C}_{k}$ if and only if $\mathbf{v}_{n}$ is generated from the $k$-th component. One of the most important goals of the learning of a mixture model is to find the correct target clustering of the samples (and thereby inferring the parameters of the model). The learning of mixture models, especially Gaussian mixture models (GMMs) [1, 2], is of fundamental importance.

Clustering is a ubiquitous problem in various applications, such as analyzing the information contained in gene expression data and performing market research according to firms’ financial characteristics. Objective-based clustering is a commonly-used technique for clustering. This is the procedure of minimizing a certain objective function to partition data samples into a fixed or appropriately-selected number of subsets known as clusters. The $k$-means algorithm [3] is perhaps the most popular objective-based clustering approach. Suppose we have a data matrix of $N$ samples $\mathbf{V}=[\mathbf{v}_{1},\mathbf{v}_{2},\ldots,\mathbf{v}_{N}]\in\mathbb{R}^{F\times N}$, a $K$-clustering (or simply a clustering or a partition) is defined as a set of pairwise disjoint index sets $\mathscr{C}:=\{\mathscr{C}_{1},\mathscr{C}_{2},\ldots,\mathscr{C}_{K}\}$ whose union is $\{1,2,\ldots,N\}$. The corresponding sum-of-squares distortion measure with respect to $\mathbf{V}$ and $\mathscr{C}$ is defined as

[TABLE]

where $\mathbf{c}_{k}:=\frac{1}{|\mathscr{C}_{k}|}\sum_{n\in\mathscr{C}_{k}}\mathbf{v}_{n}$ is the cluster center or centroid of the $k$-th cluster. The goal of the $k$-means algorithm is to find an optimal clustering $\mathscr{C}^{\mathrm{opt}}$ that satisfies

[TABLE]

where the minimization is taken over all $K$-clusterings. Optimizing this objective function is NP-hard [4]. Despite the wide usage of $k$-means and some analysis of the $k$-means algorithm [5, 6, 7], there are relatively fewer theoretical investigations of the fundamental or information-theoretic performance limits of optimal clusterings obtained via optimizing (1). In real applications, such as clustering face images by identities, there are certain unknown correct target clusterings. While using $k$-means as a clustering algorithm, we make a key implicit assumption that any optimal clustering is close to the correct target clustering [8]. If there is an optimal clustering that is far away from the correct target clustering, then using $k$-means is meaningless because even if we obtain an optimal or approximately-optimal clustering, it may not be close to the desired correct target clustering.

Furthermore, due to the inherent inefficiencies in processing high-dimensional data, dimensionality reduction has received considerable attention. Applying such techniques before clustering high-dimensional datasets can lead to significantly faster running times and reduced memory sizes. In addition, algorithms for learning GMMs usually include a dimensionality reduction step. For example, Dasgupta [9] shows that general ellipsoidal Gaussians become “more spherical” and thereby more amenable to (successful) analysis after a random projection onto a low-dimensional subspace. Vempala and Wang [10] show that reducing dimensionality by spectral decomposition leads to the amplification of the separation among Gaussian components.

I-A Main Contributions

There are three main contributions in this paper, all concerned with fundamental limits of optimal solutions to (2).

We prove that if the data points are independently generated from a $K$-component spherical GMM with an appropriate separability assumption and the so-called non-degeneracy condition [11, 12] (see Definition 1 to follow), then any optimal clustering of the data points is close to the correct target clustering with high probability provided the number of samples is commensurately large. We extend this result to mixtures of log-concave distributions. 2. 2.

We prove that under the same data generation process, if the data points are projected onto a low-dimensional space using the first $K-1$ principal components of the empirical covariance matrix, then, under similar conditions, any optimal clustering for the data points with reduced dimensionality is close to the correct target clustering with high probability. Again, this result is extended to mixtures of log-concave distributions. 3. 3.

We show that under appropriate conditions, any approximately-optimal clustering of the data points is close to the correct target clustering. This enables us to use the theoretical framework provided herein to analyze various dimensionality reduction techniques such as random projection and randomized singular value decomposition (SVD). It also allows us to combine our theoretical analyses with efficient variants of $k$-means that return approximately-optimal clusterings.

I-B Notations

We use upper and lower case boldface letters to denote matrices and vectors respectively. We use $\mathrm{diag}(\mathbf{w})$ to represent the diagonal matrix whose diagonal entries are given by $\mathbf{w}$. The Frobenius and spectral norms of $\mathbf{V}$ are written as $\|\mathbf{V}\|_{\mathrm{F}}$ and $\|\mathbf{V}\|_{2}$ respectively. We use $\mathrm{tr}(\mathbf{X})$ to represent the trace of matrix $\mathbf{X}$. The matrix inner product between $\mathbf{A}$ and $\mathbf{B}$ is written as $\langle\mathbf{A},\mathbf{B}\rangle:=\mathrm{tr}(\mathbf{A}^{T}\mathbf{B})$. We write $[N]=\{1,2,\cdots,N\}$. Let $\mathbf{V}_{1}\in\mathbb{R}^{F\times N_{1}}$ and $\mathbf{V}_{2}\in\mathbb{R}^{F\times N_{2}}$, we denote by $\left[\mathbf{V}_{1},\mathbf{V}_{2}\right]$ the horizontal concatenation of the two matrices. We use $\mathbf{V}(1\colon I,\colon)$ to denote the first $I$ rows of $\mathbf{V}$ and use $\mathbf{V}(\colon,1\colon J)$ to denote the first $J$ columns of $\mathbf{V}$; in particular, $\mathbf{V}(i,j)$ is the element in the $(i,j)^{\mathrm{th}}$ position of $\mathbf{V}$. The SVD of a symmetric matrix $\mathbf{A}\in\mathbb{R}^{F\times F}$ is given by $\mathbf{A}=\mathbf{U}\mathbf{D}\mathbf{U}^{T}$ with $\mathbf{U}\in\mathbb{R}^{F\times F}$ being an orthogonal matrix and $\mathbf{D}\in\mathbb{R}^{F\times F}$ being a diagonal matrix. In addition, when $R:=\mathrm{rank}(\mathbf{A})<F$, the so-called compact SVD of $\mathbf{A}$ is $\mathbf{A}=\mathbf{U}_{R}\mathbf{D}_{R}\mathbf{U}_{R}^{T}$, where $\mathbf{U}_{R}:=\mathbf{U}(\colon,1\colon R)$ and $\mathbf{D}_{R}:=\mathbf{\Sigma}(1\colon R,1\colon R)$. The centering of any data matrix $\mathbf{V}=[\mathbf{v}_{1},\mathbf{v}_{2},\ldots,\mathbf{v}_{N}]$ is a shift with respect to the mean vector $\bar{\mathbf{v}}=\frac{1}{N}\sum_{n=1}^{N}\mathbf{v}_{n}$ and the resultant data matrix $\mathbf{Z}=[\mathbf{z}_{1},\mathbf{z}_{2},\ldots,\mathbf{z}_{N}]$, with $\mathbf{z}_{n}=\mathbf{v}_{n}-\bar{\mathbf{v}}$ for $n\in[N]$, is said to be the centralized matrix of $\mathbf{V}$. For a $K$-component mixture model and for any $k\in[K]$, we always use $\mathbf{u}_{k}$ to denote the component mean vector, and use $\mathbf{\Sigma}_{k}$ to denote the component covariance matrix. When $\mathbf{\Sigma}_{k}=\sigma_{k}^{2}\mathbf{I}$ for all $k\in[K]$, where $\mathbf{I}$ is the identity matrix, we say the mixture model is spherical and $\sigma_{k}^{2}$ is the variance of the $k$-th component.

I-C Paper Outline

This paper is organized as follows. In Section II, we mention some related works on learning mixture models, $k$-means clustering, and dimensionality reduction. Our main theoretical results concerning upper bounds on the misclassification error distance for spherical GMMs are presented in Section III. These results are extended to mixtures of log-concave distributions in Section IV. Other extensions using other dimensionality-reduction techniques and the combination of our results with efficient clustering algorithms are discussed in Section V. We conclude our discussion and suggest avenues for future research in Section VI. Proofs are relegated to the various appendices.

II Related Work

In this section, we discuss some relevant existing works.

II-A Learning (Gaussian) Mixture Models

The learning of GMMs is of fundamental importance in machine learning and its two most important goals are: (i) Inferring the parameters of the GMM; (ii) Finding the correct target clustering of data samples according to which Gaussian distribution they were generated from. The EM algorithm [13, 14] is widely used to estimate the parameters of a GMM. However, EM is a local-search heuristic approach for maximum likelihood estimation in the presence of incomplete data and in general, it cannot guarantee the parameters’ convergence to global optima [15]. Recently, Hsu and Kakade [11] and Anandkumar et al. [12] provided approaches based on spectral decomposition to obtain consistent parameter estimates for spherical GMMs from first-, second- and third-order observable moments. To estimate parameters, they need to assume the so-called non-degeneracy condition for spherical GMMs with parameters $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$.

Definition 1**.**

(Non-degeneracy condition) We say that a mixture model satisfies the non-degeneracy condition if its component mean vectors $\mathbf{u}_{1},\ldots,\mathbf{u}_{K}$ span a $K$-dimensional subspace and the probability vector $\mathbf{w}$ has strictly positive entries.

On the other hand, under certain separability assumptions on the Gaussian components, Dasgupta [9], Dasgupta and Schulman [16], Arora and Kannan [17], Vempala and Wang [10] and Kalai et al. [18] provided provably correct algorithms that guarantee that most samples are correctly classified or that parameters are recovered with a certain accuracy with high probability. In particular, equipped with the following separability assumption

[TABLE]

for a spherical GMM, where $C>0$ is a sufficiently large constant111Throughout, we use the generic notations $C$ and $C_{i},i\in\mathbb{N}$ to denote (large) positive constants. They depend on the parameters of the mixture model and may change from usage to usage. and $w_{\min}:=\min_{k\in[K]}w_{k}$, Vempala and Wang [10] present a simple spectral algorithm with running time polynomial in both $F$ and $K$ that correctly clusters random samples according to which spherical Gaussian they were generated from.

Despite the large number of algorithms designed to find the (approximately) correct target clustering of a GMM, many practitioners use $k$-means because of its simplicity and successful applications in various fields. Kumar and Kannan [19] show that the $k$-means algorithm with a proper initialization can correctly cluster nearly all the data points generated from a GMM that satisfies a certain proximity assumption. Our theoretical results provide an explanation on why the $k$-means algorithm that attempts to find an optimal clustering is a good choice for learning mixture models. We compare and contrast our work to that of Kumar and Kannan [19] in Remark 5.

II-B Lower Bound on Distortion and the ME Distance

Let $\mathbf{V}=[\mathbf{v}_{1},\mathbf{v}_{2},\ldots,\mathbf{v}_{N}]\in\mathbb{R}^{F\times N}$ be a dataset and $\mathscr{C}:=\{\mathscr{C}_{1},\mathscr{C}_{2},\ldots,\mathscr{C}_{K}\}$ be a $K$-clustering. Let $\mathbf{H}\in\mathbb{R}^{K\times N}$ with elements $\mathbf{H}(k,n)$ be the clustering membership matrix satisfying $\mathbf{H}(k,n)=1$ if $n\in\mathscr{C}_{k}$ and $\mathbf{H}(k,n)=0$ if $n\notin\mathscr{C}_{k}$ for $(k,n)\in[K]\times[N]$. Let $n_{k}=|\mathscr{C}_{k}|$ and $\bar{\mathbf{H}}:=\mathrm{diag}(\frac{1}{\sqrt{n_{1}}},\frac{1}{\sqrt{n_{2}}},\ldots,\frac{1}{\sqrt{n_{K}}})\mathbf{H}$ be the normalized version of $\mathbf{H}$. We have $\bar{\mathbf{H}}\bar{\mathbf{H}}^{T}=\mathbf{I}$ and the corresponding distortion can be written as [20]

[TABLE]

Let $\mathbf{Z}$ be the centralized data matrix of $\mathbf{V}$ and define $\mathbf{S}:=\mathbf{Z}^{T}\mathbf{Z}$. Note that $\mathcal{D}(\mathbf{V},\mathscr{C})=\mathcal{D}(\mathbf{Z},\mathscr{C})$ for any clustering $\mathscr{C}$. Ding and He [20] make use of this property to provide a lower bound $\mathcal{D}^{*}(\mathbf{V})$ for distortion over all $K$-clusterings. That is, for any $K$-clustering $\mathscr{C}$,

[TABLE]

where $\lambda_{1}(\mathbf{S})\geq\lambda_{2}(\mathbf{S})\geq\ldots\geq 0$ are the eigenvalues of $\mathbf{S}$ sorted in non-increasing order.

For any two $K$-clusterings, the so-called misclassification error (ME) distance provides a quantitative comparison of their structures.

Definition 2**.**

(ME distance) The misclassification error distance of any two $K$-clusterings $\mathscr{C}^{(1)}:=\{\mathscr{C}_{1}^{(1)},\mathscr{C}_{2}^{(1)},\ldots,\mathscr{C}_{K}^{(1)}\}$ and $\mathscr{C}^{(2)}:=\{\mathscr{C}_{1}^{(2)},\mathscr{C}_{2}^{(2)},\ldots,\mathscr{C}_{K}^{(2)}\}$ is

[TABLE]

where $\mathcal{P}_{K}$ is the set of all permutations of $[K]$. It is known from Meilă [21] that the ME distance is indeed a distance.

For any $\delta,\delta^{\prime}\in[0,K-1]$, define the functions

[TABLE]

Combining Lemma 2 and Theorem 3 in Meilă [22], we have the following lemma.

Lemma 1**.**

Let $\mathscr{C}:=\{\mathscr{C}_{1},\mathscr{C}_{2},\ldots,\mathscr{C}_{K}\}$ and $\mathscr{C}^{\prime}:=\{\mathscr{C}^{\prime}_{1},\mathscr{C}^{\prime}_{2},\ldots,\mathscr{C}^{\prime}_{K}\}$ be two $K$-clusterings of a dataset $\mathbf{V}\in\mathbb{R}^{F\times N}$. Let $p_{\max}:=\max_{k}\frac{1}{N}|\mathscr{C}_{k}|$ and $p_{\min}:=\min_{k}\frac{1}{N}|\mathscr{C}_{k}|$. Let $\mathbf{Z}$ be the centralized matrix of $\mathbf{V}$ and $\mathbf{S}=\mathbf{Z}^{T}\mathbf{Z}$. Define

[TABLE]

Then if $\delta,\delta^{\prime}\leq\frac{1}{2}(K-1)$ and $\tau(\delta,\delta^{\prime})\leq p_{\min}$, we have

[TABLE]

This lemma says that any two “good” $K$-clusterings (in the sense that their distortions are sufficiently close to the lower bound of distortion ${\cal D}^{*}(\mathbf{V})$) are close to each other. In addition, we have the following useful corollary.

Corollary 1**.**

Let $\mathscr{C}:=\{\mathscr{C}_{1},\mathscr{C}_{2},\ldots,\mathscr{C}_{K}\}$ be a $K$-clustering of a dataset $\mathbf{V}\in\mathbb{R}^{F\times N}$ and define $p_{\max}$, $p_{\min}$, $\mathbf{Z}$, $\mathbf{S}$, and $\delta$ as in Lemma 1. Then if $\delta\leq\frac{1}{2}(K-1)$ and $\tau(\delta)\leq p_{\min}$, we have

[TABLE]

where $\mathscr{C}^{\mathrm{opt}}$ represents a $K$-clustering that minimizes the distortion for $\mathbf{V}$.

This corollary essentially says that if the distortion of a clustering is sufficiently close to the lower bound of distortion, then this clustering is close to any optimal clustering with respect to the ME distance.

II-C Principal Component Analysis

Principal component analysis (PCA) [23] is a popular strategy to compute the directions of maximal variances in vector-valued data and is widely used for dimensionality reduction. For any dataset $\mathbf{V}\in\mathbb{R}^{F\times N}$ and any positive integer $k\leq F$, the so-called $k$-PCA for the dataset usually consists of two steps: (i) Obtain the centralized dataset $\mathbf{Z}$; (ii) Calculate the SVD of $\bar{\mathbf{\Sigma}}_{N}:=\frac{1}{N}\mathbf{Z}\mathbf{Z}^{T}$, i.e., obtain $\bar{\mathbf{\Sigma}}_{N}=\mathbf{P}\mathbf{D}\mathbf{P}^{T}$, and project the dataset onto a $k$-dimensional space to obtain $\tilde{\mathbf{V}}:=\mathbf{P}_{k}^{T}\mathbf{V}$, where $\mathbf{P}_{k}:=\mathbf{P}(\colon,1\colon k)$. For brevity, we say that $\tilde{\mathbf{V}}$ is the post-$k$-PCA dataset of $\mathbf{V}$ (or simply the post-PCA dataset). If only the projection step is performed (and not the centralizing step), we term the corresponding approach $k$-PCA with no centering or simply $k$-SVD, and we say that the corresponding $\tilde{\mathbf{V}}$ is the post-$k$-SVD dataset (or simply the post-SVD dataset) of $\mathbf{V}$.

II-D Comparing Optimal Clusterings for the Original Dataset and the Post-PCA Dataset

When performing dimensionality reduction for clustering, it is important to compare any optimal clustering for the dataset with reduced dimensionality to any optimal clustering for the original dataset. More specifically, any optimal clustering for the dataset with reduced dimensionality should be close to any optimal clustering for the original dataset. However, existing works [24, 25] that combine $k$-means clustering and dimensionality reduction can only guarantee that the distortion of any optimal clustering for the dataset with reduced dimensionality, $\tilde{\mathscr{C}}^{\mathrm{opt}}$, can be bounded by a factor $\gamma>1$ times the distortion of any optimal clustering for the original dataset, $\mathscr{C}^{\mathrm{opt}}$. That is,

[TABLE]

As mentioned in Boutsidis et al. [24], directly comparing the structures of $\tilde{\mathscr{C}}^{\mathrm{opt}}$ and $\mathscr{C}^{\mathrm{opt}}$ (instead of their distortions) is more interesting. In this paper, we also prove that, if the samples are generated from a spherical GMM (or a mixture of log-concave distributions) which satisfies a separability assumption and the non-degeneracy condition, when the number of samples is sufficiently large, the ME distance between any optimal clustering for the original dataset and any optimal clustering for the post-PCA dataset can be bounded appropriately.

In addition, we can show that any optimal clustering of the dimensionality-reduced dataset is close to the correct target clustering by leveraging (12). This simple strategy seems to be adequate for data-independent dimensionality reduction techniques such as random projection. However, for data-dependent dimensionality reduction techniques such as PCA, we believe that it is worth applying distinct proof techniques similar to those developed herein to obtain stronger theoretical results because of the generative models we assume. See Section V-A for a detailed discussion.

III Results for Spherical GMMs

In this section, we assume the datasets are generated from spherical GMMs. Even though we can and will make statements for more general log-concave distributions (see Section IV), this assumption allows us to illustrate our results and mathematical ideas as cleanly as possible. We first present our main theorems for the upper bounds of ME distance between any optimal clustering and the correct target clustering for both the original and the dimensionality-reduced datasets in Section III-A. We augment our results by several remarks and illustrative examples. In Section III-B, we conduct some numerical experiments to verify the correctness and tightness of the bounds presented in Section III-A. In Section III-C, we provide several useful lemmas. Finally, in Sections III-D and III-E, we provide our proofs (or proof sketches) for the theorems.

III-A *Description of the Theorems *

First, we show that with the combination of a new separability assumption and the non-degeneracy condition (cf. Definition 1) for a spherical GMM, any optimal clustering for a dataset generated from the spherical GMM is close to the correct target clustering with high probability when the number of samples is sufficiently large.

We adopt the following set of notations. Let $\mathbf{\Sigma}_{N}:=\frac{1}{N}\mathbf{V}\mathbf{V}^{T}$ and $\bar{\mathbf{\Sigma}}_{N}:=\frac{1}{N}\mathbf{Z}\mathbf{Z}^{T}$, where $\mathbf{Z}$ is the centralized matrix of $\mathbf{V}$. Fix a mixture model with parameters $\{(w_{k},\mathbf{u}_{k},\mathbf{\Sigma}_{k})\}_{k\in[K]}$ where $w_{k}$, $\mathbf{u}_{k}$ and $\mathbf{\Sigma}_{k}$ denote the mixing weight, the mean vector, and the covariance matrix of the $k$-th component. Let

[TABLE]

Denote $\bar{\mathbf{u}}:=\sum_{k=1}^{K}w_{k}\mathbf{u}_{k}$ and write

[TABLE]

and $\lambda_{\min}:=\lambda_{K-1}(\bar{\mathbf{\Sigma}}_{0})$. For a $K$-component spherical mixture model with covariance matrices $\sigma_{k}^{2}\mathbf{I}$ for $k\in[K]$, we write $\bar{\sigma}^{2}:=\sum_{k=1}^{K}w_{k}\sigma_{k}^{2}$.

For $p\in[0,\frac{1}{2}(K-1)]$, we define the function

[TABLE]

We clearly have $\frac{1}{2}p\leq\zeta(p)\leq p$. Our first theorem reads:

Theorem 1**.**

Suppose all the columns of data matrix $\mathbf{V}\in\mathbb{R}^{F\times N}$ are independently generated from a $K$-component spherical GMM and $N>F>K$. Assume the spherical GMM satisfies the non-degeneracy condition (cf. Definition 1). Let $w_{\min}:=\min_{k}w_{k}$ and $w_{\max}:=\max_{k}w_{k}$. Further assume that

[TABLE]

Let $\mathscr{C}:=\{\mathscr{C}_{1},\mathscr{C}_{2},\ldots,\mathscr{C}_{K}\}$ be the correct target $K$-clustering corresponding to the spherical GMM. Assume that $\epsilon>0$ satisfies

[TABLE]

Then for any $t\geq 1$, if $N\geq CF^{5}K^{2}t^{2}/\epsilon^{2}$, where $C>0$ depends on $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$, we have, with probability at least $1-36KF^{2}\exp\left(-t^{2}F\right)$,

[TABLE]

where $\mathscr{C}^{\mathrm{opt}}$ is an optimal $K$-clustering for $\mathbf{V}$ and $\tau(\cdot)$ is defined in (8).

The proof of Theorem 1 is based on Corollary 1 and various concentration bounds. A sketch is provided in Section III-D while the complete proof is provided in Appendix -A. Note that both $\tau(\mathord{\cdot})$ and $\zeta(\mathord{\cdot})$ are continuous and monotonically increasing on $[0,\frac{1}{2}(K-1)]$. When the mixing weights are skewed (leading to a small $w_{\min}$), we require a strong separability assumption in (17). This is consistent with Xiong et al. [26] which suggests that imbalanced clusters are more difficult to disambiguate for $k$-means. When $\delta_{0}$ is small (i.e., the data is well-separated) and $N$ is large (so $\epsilon$ and $t$ can be chosen to be sufficiently small and large respectively), we have with probability close to $1$ that the upper bound on the ME distance given by (18) is close to [math].

When the ME distance between any optimal clustering for $k$-means $\mathscr{C}^{\mathrm{opt}}$ and the correct target clustering $\mathscr{C}$ of the samples generated from a spherical GMM is small (and thus the implicit assumption of $k$-means is satisfied), we can readily perform $k$-means to find $\mathscr{C}^{\mathrm{opt}}$ to infer $\mathscr{C}$. The tightness of the upper bound in (18) is assessed numerically in Section III-B.

Remark 1**.**

The condition (16) can be considered as a separability assumption. In particular, when $K=2$, we have $\lambda_{\min}=w_{1}w_{2}\|\mathbf{u}_{1}-\mathbf{u}_{2}\|_{2}^{2}$ and (16) becomes

[TABLE]

which is similar to (3), the separability assumption of Vempala and Wang [10]. Unlike (3), in our separability assumption, there is no dependence on $F$, which is an advantage for datasets with a large number of features, such as the outputs of deep neural networks. There are also no implicit constants such as $C$ in (3).**

Remark 2**.**

For general $K\geq 2$, we consider the following example so that the separability condition in (16) and the bound on the ME distance in (18) become more transparent.

• •

Assume that $w_{i}={1}/{K}$ for all $i\in[K]$;

• •

Assume that there is a positive number $d$ such that $\|\mathbf{u}_{i}-\bar{\mathbf{u}}\|_{2}=d$ for all $i\in[K]$ and there is an angle $\alpha\in[0,\pi]$ such that $(\mathbf{u}_{i}-\bar{\mathbf{u}})^{T}(\mathbf{u}_{j}-\bar{\mathbf{u}})=d^{2}\cos\alpha$ for all $i\neq j$.

Then for this rather symmetrical setting, we have that

• •

$\cos\alpha=-{1}/(K-1)$;

• •

$\|\mathbf{u}_{i}-\mathbf{u}_{j}\|_{2}^{2}=2d^{2}(1-\cos\alpha)=2d^{2}{K}/(K-1)$ for all $i\neq j$;

• •

$\lambda_{\min}=\lambda_{K-1}(\bar{\bm{\Sigma}}_{0})=d^{2}/(K-1)={\|\mathbf{u}_{i}-\mathbf{u}_{j}\|_{2}^{2}}/(2K)$ for all $i\neq j$.

If we further consider the relaxation $\zeta(w_{\min})>{w_{\min}}/{2}$ (this is due to the definition of $\zeta(\cdot)$ in (15)), the separability condition (16) can be written as

[TABLE]

or equivalently,

[TABLE]

For this special case, compared to the separability condition (3) provided by Vempala and Wang [10], there is no dependence on $F$ in our separability condition in (21). Moreover, (21) matches the separability condition given by Kumar and Kannan [19] which is $\Omega({K\bar{\sigma}}/{\sqrt{w_{\min}}})$. However, note that the settings in [19] and our paper are significantly different. Kumar and Kannan [19] analyze a variant of the $k$-means algorithm, whilst we analyze optimal solutions of the sums-of-squares objective function (also see Remark 5). It is thus reassuring that these two considerations lead to the same type of separability condition. Furthermore, note that for this simple example,

[TABLE]

If we leverage the inequality $\tau(\delta)<2\delta$ (which follows from the definition of $\tau(\cdot)$ in (8)) and ignore the $\epsilon$ term in the upper bound provided in (18), we have that the upper bound for ME distance becomes

[TABLE]

The upper bound in (23) has an intuitive flavor. It is proportional to $\bar{\sigma}^{2}$ (a measure of the spread of the intra-cluster points) and inversely proportional to $d^{2}$ (the squared distance between the component mean vectors and their average). Similar separability conditions and upper bounds for ME distance can be obtained for this example in the theorems to follow.**

Remark 3**.**

The separability condition in (16) is different from some other pairwise separability assumptions in the literature [9, 16, 17, 10, 18]. Our condition is a global separability condition. The intuitive reasons for this are twofold. First, we study the optimal solutions to the sum-of-squares distortion measure in (1). This is a global measure, involving all clusters, and so we believe a global separability condition is natural. Second, we leverage several technical lemmas in the literature, such as Lemma 1. These lemmas also involve global parameters such as $\lambda_{K-1}(\mathbf{S})$ and $\lambda_{K}(\mathbf{S})$, thus a global separability condition of the form of (16) seems unavoidable.**

Remark 4**.**

The non-degeneracy condition is used to ensure that $\lambda_{\min}>0$. When $K=2$, to ensure that $\lambda_{\min}>0$, from (19), we see that we only need to assume that the two component mean vectors are distinct. In particular, we do not require $\mathbf{u}_{1}$ and $\mathbf{u}_{2}$ to be linearly independent.**

Remark 5**.**

The result by Kumar and Kannan [19] (discussed in Section II-A) may, at a first glance, appear to be similar to Theorem 1 in the sense that both results show that under appropriate conditions, the $k$-means algorithm is a good choice for learning certain mixture models. However, there is a salient difference. The analysis of Kumar and Kannan [19] is based on a variant of the $k$-means algorithm, while we only analyze the objective function of $k$-means (in (1)) which determines all optimal clusterings. Since there are multiple ways to approximately minimize the ubiquitous but intractable sum-of-squares distortion measure in (1), our analysis is partly algorithm-independent (but dependent on the objective function in (1)) and thus fundamental in the theory of clustering. Our analysis and theoretical results, in fact, underpin why the separability assumptions of various forms appear to be necessary to make theoretical guarantees for using $k$-means to learn mixture models.**

Next, we show that under similar assumptions for the generating process and with a weaker separability assumption for the spherical GMM, any optimal clustering for the post-PCA dataset is also close to the correct target clustering with high probability when $N$ is large enough.

Theorem 2**.**

Let the dataset $\mathbf{V}\in\mathbb{R}^{F\times N}$ be generated under the same conditions given in Theorem 1 with the separability assumption (16) being modified to

[TABLE]

Let $\tilde{\mathbf{V}}\in\mathbb{R}^{(K-1)\times N}$ be the post-$(K-1)$-PCA dataset of $\mathbf{V}$. Then, for any $\epsilon>0$ that satisfies

[TABLE]

and for any $t\geq 1$, when $N\geq CF^{3}K^{5}t^{2}/\epsilon^{2}$, where $C>0$ depends on $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$, we have, with probability at least $1-165KF\exp\left(-t^{2}K\right)$,

[TABLE]

where $\mathscr{C}$ is the correct target clustering and $\tilde{\mathscr{C}}^{\mathrm{opt}}$ is an optimal $K$-clustering for $\tilde{\mathbf{V}}$.

The proof of Theorem 2 is provided in Section III-E.

Remark 6**.**

Vempala and Wang [10] analyzed the SVD of $\mathbf{\Sigma}_{N}$ (corresponding to PCA with no centering) instead of $\bar{\mathbf{\Sigma}}_{N}$ (corresponding to PCA). The proof of Corollary 3 in Vempala and Wang [10] is based on the key observation that the subspace spanned by the first $K$ singular vectors of $\mathbf{\Sigma}_{N}$ lies close to the subspace spanned by the $K$ component mean vectors of the spherical GMM with high probability when $N$ is large. By performing $K$-SVD (cf. Section II-C) on $\mathbf{\Sigma}_{N}$, we have the following corollary. **

Corollary 2**.**

Let the dataset $\mathbf{V}\in\mathbb{R}^{F\times N}$ be generated under the same conditions given in Theorem 1. Let $\tilde{\mathbf{V}}$ be the post-$K$-SVD dataset of $\mathbf{V}$, then for any positive $\epsilon$ satisfying (17) and for any $t\geq 1$, if $N\geq CF^{3}K^{5}t^{2}/\epsilon^{2}$, then with probability at least $1-167KF\exp\left(-t^{2}K\right)$, the same upper bound in (18) holds for $\mathrm{d}_{\mathrm{ME}}(\mathscr{C},\tilde{\mathscr{C}}^{\mathrm{opt}})$, where $\tilde{\mathscr{C}}^{\mathrm{opt}}$ is an optimal $K$-clustering for $\tilde{\mathbf{V}}$.

Combining the results of Theorems 1 and 2, by the triangle inequality for ME distance, we obtain the following corollary concerning an upper bound for $\mathrm{d}_{\mathrm{ME}}(\mathscr{C}^{\mathrm{opt}},\tilde{\mathscr{C}}^{\mathrm{opt}})$, the ME distance between any optimal clustering of the original dataset and any optimal clustering of the post-PCA dataset.

Corollary 3**.**

Let the dataset $\mathbf{V}\in\mathbb{R}^{F\times N}$ be generated under the same conditions given in Theorem 1. Let $\tilde{\mathbf{V}}$ be the post-$(K-1)$-PCA dataset of $\mathbf{V}$, then for any positive $\epsilon$ satisfying (17) and for any $t\geq 1$, if $N\geq CF^{5}K^{5}t^{2}/\epsilon^{2}$, then with probability at least $1-201KF^{2}\exp\left(-t^{2}K\right)$, $\mathrm{d}_{\mathrm{ME}}(\mathscr{C}^{\mathrm{opt}},\tilde{\mathscr{C}}^{\mathrm{opt}})$ is upper bounded by the sum of the right-hand-sides of (18) and (26).

The proof of Theorem 2 hinges mainly on the fact that the subspace spanned by the first $K-1$ singular vectors of $\bar{\mathbf{\Sigma}}_{N}$ is “close” to the subspace spanned by the first $K-1$ singular vectors of $\bar{\mathbf{\Sigma}}_{0}$. See Lemma 9 to follow for a precise statement. Note that the assumption in (24) is weaker than (16) and the upper bound given by (26) is smaller than that in (18) (if all the parameters are the same). In addition, when $K=2$, by applying PCA to the original dataset as described in Theorem 2, we obtain a $1$-dimensional dataset, which is easier to cluster optimally compared to the $2$-dimensional dataset obtained by performing PCA with no centering as described in Remark 6. These comparisons also provide a theoretical basis for the fact that centering can result in a stark difference in PCA.

III-B Verifying Theorems 1 and 2 numerically

To verify the correctness and tightness of the upper bounds given in Theorems 1 and 2 and the efficacy of clustering post-PCA samples, we perform numerical experiments on synthetic datasets. We sample data points from two types of $2$-component spherical GMMs. The dimensionality of the data points is $F=100$, and the number of samples $N$ ranges from $1000$ to $10000$. Component mean vectors are randomly and uniformly picked from the hypercube $[0,1]^{F}$. Equal mixing weights are assigned to the components. After fixing the mixing weights and the component mean vectors, $\lambda_{\min}$ is fixed. For all $k\in[K]$, we set the variances to be

[TABLE]

where $\varepsilon=10^{-6}$. In all figures, left and right plots correspond to (27) and (28) respectively.

We observe from Figure 1 that for (27), the clusters are well-separated, while for (28), the clusters are moderately well-separated. For both cases, the separability assumption (16) is satisfied. Similar to that in Boutsidis et al. [24], we use the command kmeans(V’, K, ‘Replicates’, 10, ‘MaxIter’, 1000) in Matlab to obtain an approximately-optimal clustering of $\mathbf{V}$. Here V’ represents the transpose of $\mathbf{V}$. This command means that we run $k$-means for 10 times with distinct initializations and pick the best outcome. For each run, the maximal number of iterations is set to be 1000. Define $\mathrm{d_{org}}:=\mathrm{d}_{\mathrm{ME}}(\mathscr{C},\mathscr{C}^{\mathrm{opt}})$ and define the (expected) upper bound for $\mathrm{d}_{\mathrm{ME}}(\mathscr{C},\mathscr{C}^{\mathrm{opt}})$ as $\mathrm{\bar{d}_{org}}:=\tau(\delta_{0})w_{\max}$ (provided by Theorem 1). Similarly, we define $\mathrm{d_{pca}}:=\mathrm{d}_{\mathrm{ME}}(\mathscr{C},\tilde{\mathscr{C}}^{\mathrm{opt}})$ and the (expected) upper bound for $\mathrm{d}_{\mathrm{ME}}(\mathscr{C},\tilde{\mathscr{C}}^{\mathrm{opt}})$ is defined as $\mathrm{\bar{d}_{pca}}:=\tau(\delta_{1})w_{\max}$ (given by Theorem 2). We use a superscript “$\mathrm{emp}$” to represent the corresponding empirical value. For example, $\delta_{0}^{\mathrm{emp}}:=\frac{\mathcal{D}(\mathbf{V},\mathscr{C})-\mathcal{D}^{*}(\mathbf{V})}{\lambda_{K-1}(\mathbf{S})-\lambda_{K}(\mathbf{S})}$ is an approximation of $\delta_{0}$ (calculated from the samples), and $\mathrm{\bar{d}_{org}^{emp}}:=\tau(\delta_{0}^{\mathrm{emp}})p_{\max}$ is an approximation of $\mathrm{\bar{d}_{org}}$, where $p_{\max}:=\max_{k}\frac{1}{N}|\mathscr{C}_{k}|$.

Our numerical results are reported in Figure 2. We observe that the empirical values of upper bounds are close to the corresponding expected upper bounds. This observation verifies the correctness of the probabilistic estimates. For the well-separated case in (27), we observe that the upper bounds for ME distance are small compared to the moderately well-separated case in (28). For the former, the true distances $\mathrm{d_{org}}$ and $\mathrm{d_{pca}}$ are both close to [math], even when the number of samples is 1000 (a small number in this scenario). The $k$-means algorithm can easily find an approximately-optimal clustering, which is also the approximately-correct clustering. For the moderately well-separated case, we observe that the upper bounds given in Theorem 1 and Theorem 2 are informative. In particular, they are only approximately $2.5$ times the corresponding true distances.

From Figure 3, we observe that performing $k$-means for the original (high-dimensional) datasets is significantly slower than performing $k$-means for the corresponding post-PCA datasets (the reported running times for post-PCA datasets are the sums of the running times for performing PCA and for performing $k$-means on the post-PCA datasets) when the number of samples is large. This difference is more pronounced for the moderately well-separated case. For this case and $N=10000$, we have an order of magnitude speed up. The running time for larger $N$ can be less than the running time for smaller $N$ because the number of iterations for $k$-means are possibly different. All the results are averaged over $10$ runs. All experiments we run on an Intel Core i7 CPU at 2.50GHz and 16GB of memory, and the Matlab version is 8.3.0.532 (R2014a).

III-C Useful Definitions and Lemmas

First, we present the following lemma from Golub and Van Loan [27] that provides an upper bound for perturbation of eigenvalues when the matrix is perturbed.

Lemma 2**.**

If $\mathbf{A}$ and $\mathbf{A}+\mathbf{E}$ are in $\mathbb{R}^{M\times M}$, then

[TABLE]

for any $m\in[M]$ with $\lambda_{m}(\mathbf{A})$ being the $m$-th largest eigenvalue of $\mathbf{A}$.

Because we will make use of the second-order moments of a mixture model, we present a simple lemma summarizing key facts.

Lemma 3**.**

Let $\mathbf{x}$ be a random sample from a $K$-component mixture model with parameters $\{(w_{k},\mathbf{u}_{k},\mathbf{\Sigma}_{k}\}_{k\in[K]}$. Then,

[TABLE]

and

[TABLE]

To apply Corollary 1, we need to ensure that $\lambda_{K-1}(\mathbf{S})-\lambda_{K}(\mathbf{S})$ which appears in the denominators of the expressions in (9) is positive. Note that if we assume all the columns of data matrix $\mathbf{V}$ are independently generated from a $K$-component spherical GMM, we have $\frac{1}{N}\left(\lambda_{K-1}(\mathbf{S})-\lambda_{K}(\mathbf{S})\right)\xrightarrow{\mathrm{p}}\lambda_{K-1}(\bar{\mathbf{\Sigma}}_{0})$, where $\xrightarrow{\mathrm{p}}$ represents convergence in probability as $N\to\infty$. Under the non-degeneracy condition, $\lambda_{K}(\mathbf{\Sigma}_{0})>0$. In addition, by the observation that $\bar{\mathbf{\Sigma}}_{0}=\mathbf{\Sigma}_{0}-\bar{\mathbf{u}}\bar{\mathbf{u}}^{T}$ and the following lemma in Golub and Van Loan [27], we have $\lambda_{\min}=\lambda_{K-1}(\bar{\mathbf{\Sigma}}_{0})\geq\lambda_{K}(\mathbf{\Sigma}_{0})>0$.

Lemma 4**.**

Suppose $\mathbf{B}=\mathbf{A}+\tau\mathbf{v}\mathbf{v}^{T}$ where $\mathbf{A}\in\mathbb{R}^{n\times n}$ is symmetric, $\mathbf{v}$ has unit $2$-norm (i.e., $\|\mathbf{v}\|_{2}=1$) and $\tau\in\mathbb{R}$. Then,

[TABLE]

Furthermore, in order to obtain our probabilistic estimates, we need to make use of the following concentration bounds for sub-Gaussian and sub-Exponential random variables. The definitions and relevant lemmas are extracted from Vershynin [28].

Lemma 5**.**

*(Hoeffding-type inequality)

A sub-Gaussian random variable $X$ is one such that $\left(\mathbb{E}|X|^{p}\right)^{1/p}\leq C\sqrt{p}$ for some $C>0$ and for all $p\geq 1$. Let $X_{1},\ldots,X_{N}$ be independent zero-mean sub-Gaussian random variables, then for every $\mathbf{a}=[a_{1},a_{2},\ldots,a_{N}]^{T}\in\mathbb{R}^{N}$ and every $t\geq 0$, it holds that*

[TABLE]

where $c>0$ is a constant.

Typical examples of sub-Gaussian random variables are Gaussian, Bernoulli, and all bounded random variables. A random vector $\mathbf{x}\in\mathbb{R}^{F}$ is called sub-Gaussian if $\mathbf{x}^{T}\mathbf{z}$ is a sub-Gaussian random variable for any deterministic vector $\mathbf{z}\in\mathbb{R}^{F}$.

Lemma 6**.**

*(Bernstein-type inequality)

A sub-Exponential random variable $X$ is one such that $\left(\mathbb{E}|X|^{p}\right)^{1/p}\leq Cp$ for some $C>0$ and for all $p\geq 1$. Let $X_{1},\ldots,X_{N}$ be independent zero-mean sub-Exponential random variables. It holds that*

[TABLE]

where $c>0$ is an absolute constant and $M>0$ is the maximum of the sub-Exponential norms222The sub-Exponential norm of a sub-Exponential random variable $X$ is defined as $\|X\|_{\Psi_{1}}:=\sup_{p\geq 1}p^{-1}\left(\mathbb{E}|X|^{p}\right)^{1/p}$. of $\{X_{i}\}_{i=1}^{N}$, i.e., $M=\max_{i\in[N]}\|X_{i}\|_{\Psi_{1}}$.

The set of sub-Exponential random variables includes those that have tails heavier than Gaussian. It is easy to see that a sub-Gaussian random variable is also sub-Exponential. The following lemma, which can be found in Vershynin [28], is straightforward.

Lemma 7**.**

A random variable $X$ is sub-Gaussian if and only if $X^{2}$ is sub-Exponential.

Using this lemma, we see that Lemma 6 also provides a concentration bound for the sum of the squares of sub-Gaussian random variables. Finally, we can estimate empirical covariance matrices by the following lemma. Note that in this lemma, we do not need to assume that the sub-Gaussian distribution $G$ in $\mathbb{R}^{F}$ has zero mean.

Lemma 8**.**

(Covariance estimation of sub-Gaussian distributions) Consider a sub-Gaussian distribution $G$ in $\mathbb{R}^{F}$ with covariance matrix $\mathbf{\Sigma}$. Define the empirical covariance matrix $\mathbf{\Sigma}_{N}:=\frac{1}{N}\sum_{n=1}^{N}\mathbf{v}_{n}\mathbf{v}_{n}^{T}$ where each $\mathbf{v}_{n}$ is an independent sample of $G$. Let $\epsilon\in(0,1)$ and $t\geq 1$. If $N\geq C(t/\epsilon)^{2}F$ for some constant $C>0$, then with probability at least $1-2\exp(-t^{2}F)$,

[TABLE]

III-D *Proof Sketch of Theorem 1 *

The general idea of the proof of Theorem 1 is to first estimate the terms defining $\delta$ in (9) probabilistically. Next, we apply Corollary 1 to show that any optimal clustering for the original data matrix is close to the correct target clustering corresponding to the spherical GMM. We provide the proof sketch below. Detailed calculations are deferred to Appendix -A.

Proof Sketch of Theorem 1.

We estimate every term in (9) for the correct target clustering $\mathscr{C}$. By Lemmas 5 and 6, we have for any $\epsilon\in(0,1)$,

[TABLE]

where $C_{1}>0$ depends on $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$. See Appendix -A for the detailed calculation of this and other inequalities in this proof sketch. In particular, for the justification of (36), see the steps leading to (-A.17). These simply involve the triangle inequality, the union bound, and careful probabilistic estimates. Furthermore, if $N\geq C_{2}F^{5}K^{2}t^{2}/\epsilon^{2}$, we have

[TABLE]

In addition, by Lemma 8, for any $t\geq 1$, if $N\geq C_{2}{F^{3}K^{2}t^{2}}/{\epsilon^{2}}$ (where $C_{2}>0$ also depends on $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$),

[TABLE]

Therefore, by the matrix perturbation inequalities in Lemma 2, when $N\geq C_{2}{F^{3}K^{2}t^{2}}/{\epsilon^{2}}$, we have

[TABLE]

and

[TABLE]

Combining these results, appealing to Corollary 1, the union bound, and the property that both $\tau(\mathord{\cdot})$ and $\zeta(\mathord{\cdot})$ are continuous and monotonically increasing functions on $[0,\frac{1}{2}(K-1)]$, we obtain (18) as desired. ∎

III-E Proof of Theorem 2

Here, we prove Theorem 2. Following the notations in Section II-C, we write $\bar{\mathbf{\Sigma}}_{N}=\mathbf{P}\mathbf{D}\mathbf{P}^{T}$, $\mathbf{P}_{K-1}=\mathbf{P}(\colon,1\colon K-1)$, and $\mathbf{P}_{-(K-1)}=\mathbf{P}(\colon,K\colon F)$. We also denote $\tilde{\mathbf{V}}=\mathbf{P}_{K-1}^{T}\mathbf{V}$ as the post-$(K-1)$-PCA dataset of $\mathbf{V}$. Instead of using $\mathbf{P}_{K-1}$ which is correlated to the samples, we consider the SVD of $\bar{\mathbf{\Sigma}}_{0}$ and project the original data matrix onto $\mathbb{R}^{K-1}$ using the first $K-1$ singular vectors of $\bar{\mathbf{\Sigma}}_{0}$. That is, if we write the SVD of $\bar{\mathbf{\Sigma}}_{0}$ as $\bar{\mathbf{\Sigma}}_{0}=\mathbf{Q}_{K-1}\mathbf{E}_{K-1}\mathbf{Q}_{K-1}^{T}$, we first analyze $\hat{\mathbf{V}}:=\mathbf{Q}_{K-1}^{T}\mathbf{V}$, before relating these results to those of $\tilde{\mathbf{V}}=\mathbf{P}_{K-1}^{T}\mathbf{V}$. We can similarly estimate the terms in (9) for the corresponding $(K-1)$-dimensional spherical GMM. Furthermore, we estimate the difference between the results obtained from projecting the original data matrix onto $\mathbb{R}^{K-1}$ using the first $K-1$ singular vectors of $\bar{\mathbf{\Sigma}}_{0}$ and the results obtained from projecting the original data matrix onto $\mathbb{R}^{K-1}$ using the columns of $\mathbf{P}_{K-1}$.

Proof of Theorem 2:.

By the non-degeneracy condition and Lemma 4, we have $\mathrm{rank}(\bar{\mathbf{\Sigma}}_{0})=K-1$. Let the compact SVD of $\bar{\mathbf{\Sigma}}_{0}$ be

[TABLE]

where $\mathbf{Q}_{K-1}\in\mathbb{R}^{F\times(K-1)}$ has orthonormal columns and $\mathbf{E}_{K-1}\in\mathbb{R}^{(K-1)\times(K-1)}$ is a diagonal matrix. Since $\mathbf{Q}_{K-1}^{T}\mathbf{Q}_{K-1}=\mathbf{I}$, by the property of Gaussians, we know if $\mathbf{x}$ is a random vector with a spherical Gaussian distribution $\mathcal{N}(\mathbf{u},\sigma^{2}\mathbf{I})$ in $\mathbb{R}^{F}$, then $\mathbf{Q}_{K-1}^{T}\mathbf{x}$ is a random vector with a spherical Gaussian distribution $\mathcal{N}(\mathbf{Q}_{K-1}^{T}\mathbf{u},\sigma^{2}\mathbf{I})$ in $\mathbb{R}^{K-1}$. Let $\hat{\mathbf{V}}:=\mathbf{Q}_{K-1}^{T}\mathbf{V}$, $\hat{\mathbf{Z}}$ be the centralized matrix of $\hat{\mathbf{V}}$ and $\hat{\mathbf{S}}:=\hat{\mathbf{Z}}^{T}\hat{\mathbf{Z}}$. Denote $\hat{\bar{\mathbf{u}}}=\mathbf{Q}_{K-1}^{T}\bar{\mathbf{u}}$ and $\hat{\mathbf{u}}_{k}=\mathbf{Q}_{K-1}^{T}\mathbf{u}_{k}$ for all $k\in[K]$. Let $\hat{\bar{\mathbf{\Sigma}}}_{0}:=\sum_{k=1}^{K}w_{k}(\hat{\mathbf{u}}_{k}-\hat{\bar{\mathbf{u}}})(\hat{\mathbf{u}}_{k}-\hat{\bar{\mathbf{u}}})^{T}$. Define $\mathbf{X}:=[\sqrt{w_{1}}(\mathbf{u}_{1}-\bar{\mathbf{u}}),\ldots,\sqrt{w_{K}}(\mathbf{u}_{K}-\bar{\mathbf{u}})]\in\mathbb{R}^{F\times K}$ and let $\hat{\mathbf{X}}:=\mathbf{Q}_{K-1}^{T}\mathbf{X}$. Select $\mathbf{Q}_{-(K-1)}\in\mathbb{R}^{F\times(F-K+1)}$ such that $[\mathbf{Q}_{K-1},\mathbf{Q}_{-(K-1)}]$ is an orthogonal matrix. We have

[TABLE]

Thus we have

[TABLE]

Then similar to (36) in the proof sketch of Theorem 1, for any $\epsilon\in(0,1)$,

[TABLE]

In addition, similar to (39) in the proof sketch of Theorem 1, for any $t\geq 1$, if $N\geq C_{4}{K^{5}t^{2}}/{\epsilon^{2}}$,

[TABLE]

where $C_{3},C_{4}>0$ depend on $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$. Note that $\frac{1}{N}\mathcal{D}^{*}(\hat{\mathbf{V}})=\frac{1}{N}\lambda_{K}(\hat{\mathbf{S}})=0.$ Thus, we only need to estimate \frac{1}{N}\big{|}\mathcal{D}(\hat{\mathbf{V}},\mathscr{C})-\mathcal{D}(\tilde{\mathbf{V}},\mathscr{C})\big{|} and \frac{1}{N}\big{|}\lambda_{K-1}(\hat{\mathbf{S}})-\lambda_{K-1}(\tilde{\mathbf{S}})\big{|}, where $\tilde{\mathbf{S}}:=\tilde{\mathbf{Z}}^{T}\tilde{\mathbf{Z}}$ and $\tilde{\mathbf{Z}}$ is the centralized matrix of $\tilde{\mathbf{V}}$. By (4) and writing $\mathbf{R}:=\mathbf{Q}_{K-1}\mathbf{Q}_{K-1}^{T}-\mathbf{P}_{K-1}\mathbf{P}_{K-1}^{T}$, we have

[TABLE]

Note that

[TABLE]

In addition, by Lemma 2 and routine calculations,

[TABLE]

Thus in (48) and (53), we need to bound $\|\mathbf{R}\|_{\mathrm{F}}$. According to (38), $\|\bar{\mathbf{\Sigma}}_{N}-\bar{\mathbf{\Sigma}}\|_{2}$ can be bounded probabilistically. By Lemma 9 to follow, the upper bound of $\|\mathbf{R}\|_{\mathrm{F}}$ can be deduced by the upper bound of $\|\bar{\mathbf{\Sigma}}_{N}-\bar{\mathbf{\Sigma}}\|_{2}$. By leveraging additional concentration bounds for sub-Gaussian and sub-Exponential distributions given in Lemmas 5 and 6, we deduce that if $N\geq C_{5}{F^{3}K^{5}t^{2}}/{\epsilon^{2}}$,

[TABLE]

where $C_{5}>0$ depends on $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$. The proofs of (54) and (55) are omitted for the sake of brevity. Combining these bounds with (44) and (45) and by using Corollary 1, we obtain (26) as desired. ∎

The following is a lemma essential for establishing upper bounds of (48) and (53) in the proof of Theorem 2. Note that if we view the Grassmannian manifold as a metric measure space, the distance between subspaces $\mathcal{E}$ and $\mathcal{F}$ can be defined as [29]

[TABLE]

where $\mathbf{P}_{\mathcal{E}}$ and $\mathbf{P}_{\mathcal{F}}$ are the orthogonal projections onto $\mathcal{E}$ and $\mathcal{F}$. Because $\mathbf{Q}_{K-1}\mathbf{Q}_{K-1}^{T}$ and $\mathbf{P}_{K-1}\mathbf{P}_{K-1}^{T}$ are the orthogonal projection matrices for projections onto the subspaces spanned by the columns of $\mathbf{Q}_{K-1}$ and $\mathbf{P}_{K-1}$ respectively, $\|\mathbf{R}\|_{\mathrm{F}}$ is a measure of the distance between these two subspaces.

Lemma 9**.**

For $\epsilon>0$, if $\|\bar{\mathbf{\Sigma}}_{N}-\bar{\mathbf{\Sigma}}\|_{2}\leq\epsilon$, then

[TABLE]

Proof.

By Lemma 2, $|\lambda_{F}(\bar{\mathbf{\Sigma}}_{N})-\bar{\sigma}^{2}|=|\lambda_{F}(\bar{\mathbf{\Sigma}}_{N})-\lambda_{F}(\bar{\mathbf{\Sigma}})|\leq\epsilon$. Then

[TABLE]

Because $\bar{\mathbf{\Sigma}}_{N}-\lambda_{F}(\bar{\mathbf{\Sigma}}_{N})\mathbf{I}$ is also positive semidefinite, the SVD is

[TABLE]

where $\bar{\mathbf{D}}:=\mathbf{D}-\lambda_{F}(\bar{\mathbf{\Sigma}}_{N})\mathbf{I}$. Let $\bar{\mathbf{D}}_{K-1}=\bar{\mathbf{D}}(1\colon K-1,1\colon K-1)$. Note that $\mathrm{rank}(\bar{\mathbf{\Sigma}}_{0})=K-1$. By Lemma 2, $\lambda_{K}(\mathbf{P}\bar{\mathbf{D}}\mathbf{P}^{T})\leq 2\epsilon$ and thus we have

[TABLE]

Therefore, there exists a matrix $\mathbf{E}_{0}$ with $\|\mathbf{E}_{0}\|_{2}\leq 4\epsilon$, such that

[TABLE]

That is,

[TABLE]

Recall that $\mathbf{P}_{-(K-1)}:=\mathbf{P}(:,K:F)$. We obtain

[TABLE]

where $\bar{\mathbf{\Sigma}}_{0}^{+}:=\mathbf{Q}_{K-1}\mathbf{E}_{K-1}^{-1}\mathbf{Q}_{K-1}^{T}$ is the Moore-Penrose generalized inverse (or pseudoinverse) of $\bar{\mathbf{\Sigma}}_{0}$ and its largest eigenvalue is $\lambda_{\min}^{-1}$. Because $\mathbf{P}_{-(K-1)}\mathbf{P}_{-(K-1)}^{T}$ projects vectors in $\mathbb{R}^{F}$ onto the linear space spanned by the orthonormal columns of $\mathbf{P}_{-(K-1)}$, we have

[TABLE]

where we use the inequality that $\|\mathbf{M}\mathbf{N}\|_{\mathrm{F}}\leq\|\mathbf{M}\|_{2}\|\mathbf{N}\|_{\mathrm{F}}$ for any two compatible matrices $\mathbf{M}$ and $\mathbf{N}$. The inequality $\|\bar{\mathbf{\Sigma}}_{0}^{+}\|_{\mathrm{F}}\leq\sqrt{K}\|\bar{\mathbf{\Sigma}}_{0}^{+}\|_{2}$ arises because $\bar{\mathbf{\Sigma}}_{0}^{+}$ only contains $K-1$ positive eigenvalues. ∎

Remark 7**.**

By Lemmas 8 and 9, we see that the subspace spanned by the first $K-1$ singular vectors of $\bar{\mathbf{\Sigma}}_{0}$ lies close to the subspace spanned by the first $K-1$ singular vectors of $\bar{\mathbf{\Sigma}}_{N}$ when the number of samples is sufficiently large. Note that $\sum_{k=1}^{K}w_{k}(\mathbf{u}_{k}-\bar{\mathbf{u}})(\mathbf{u}_{k}-\bar{\mathbf{u}})^{T}=\bar{\mathbf{\Sigma}}_{0}=\mathbf{Q}_{K-1}\mathbf{E}_{K-1}\mathbf{Q}_{K-1}^{T}$. We also obtain that the subspace spanned by $\mathbf{u}_{k}-\bar{\mathbf{u}},k\in[K]$ is close to the subspace spanned by the first $K-1$ singular vectors of $\bar{\mathbf{\Sigma}}_{N}$. Note that $\mathbf{\Sigma}_{0}=\sum_{k=1}^{K}w_{k}\mathbf{u}_{k}\mathbf{u}_{k}^{T}$. A similar result can be obtained for $K$-SVD to corroborate an observation by Vempala and Wang [10] (cf. Remark 6).**

IV Extension to Mixtures of Log-Concave Distributions

In this section, we extend the results in Section III to mixtures of log-concave distributions. This is motivated partly by Arora and Kannan [17] who mention the algorithm they design for analyzing the learning of GMMs may be extended to log-concave distributions (besides Gaussians). Also, Kannan et al. [30] generalize the work of Vempala and Wang [10] from spherical GMMs to mixtures of log-concave distributions. Furthermore, Brubaker [31] considers the robust learning of mixtures of log-concave distributions. We also consider the learning of mixtures of log-concave distributions although the structure of the theoretical analysis is mostly similar to that for spherical GMMs. We provide some necessary preliminary definitions and results for log-concave distributions and random variables in Section IV-A. To prove our main results concerning such distributions, we need to employ a slightly different proof strategy vis-à-vis the one for spherical GMMs. We discuss these differences in Section IV-B. After presenting some preliminary lemmas, we present our main theorems for mixtures of log-concave distributions in Section IV-C.

IV-A Definitions and Useful Lemmas

A function $f:\mathbb{R}^{F}\rightarrow\mathbb{R}_{+}$ is log-concave if its logarithm $\log f$ is concave. That is, for any two vectors $\mathbf{x},\mathbf{y}\in\mathbb{R}^{F}$ and any $\alpha\in[0,1]$,

[TABLE]

A distribution is log-concave if its probability density (or mass) function is log-concave. We say a random variable is log-concave if its distribution is log-concave. There are many distributions that are log-concave, including Gaussian distributions, exponential distributions, and Laplace distributions. In particular, distributions that belong to exponential families are log-concave. Log-concave distributions have several desirable properties. For example, the sum of two independent log-concave random variables (i.e., the convolution of two log-concave distributions) is also log-concave. In addition, the linear projection of a log-concave distribution onto a lower-dimensional space remains log-concave. To start off, we need to estimate the deviation of an empirical covariance matrix from the true covariance matrix. We leverage the following lemma due to Kannan et al. [30].

Lemma 10**.**

Let $\epsilon,\eta\in(0,1)$, and $\mathbf{y}_{1},\mathbf{y}_{2},\ldots,\mathbf{y}_{N}$ be zero-mean i.i.d. random vectors from a log-concave distribution in $\mathbb{R}^{F}$. Then there is an absolute constant $C>0$ such that if $N>C\frac{F}{\epsilon^{2}}\log^{5}\left(\frac{F}{\epsilon\eta}\right)$, with probability at least $1-\eta$, $\forall\,\mathbf{v}\in\mathbb{R}^{F}$,

[TABLE]

Note that Lemma 10 provides an estimate for the empirical covariance matrix. This is because that for any symmetric matrix $\mathbf{M}$, $\|\mathbf{M}\|_{2}=\sup_{\mathbf{v}\in\mathbb{R}^{F},\|\mathbf{v}\|_{2}=1}|\langle\mathbf{M}\mathbf{v},\mathbf{v}\rangle|$ and (69) is equivalent to

[TABLE]

Using Lemma 10, we also have the following corollary which provides a useful concentration bound for the sum of the squares of log-concave random variables.

Corollary 4**.**

Let $\epsilon\in(0,1)$, and $y_{1},y_{2},\ldots,y_{N}$ be i.i.d. samples from a log-concave distribution with expectation $\mu$ and variance $\sigma^{2}$. Then if $\epsilon$ is sufficiently small, there is an absolute constant $C>0$ such that for $N>C\frac{1}{\epsilon^{2}}\log^{5}\left(\frac{1}{\epsilon\eta}\right)$, with probability at least $1-\eta$,

[TABLE]

The proof of the above corollary follows directly from Lemma 10 and so it is omitted. We will make use of the following lemma [27] concerning the eigenvalues of a matrix which is the sum of two matrices.

Lemma 11**.**

If $\mathbf{A}$ and $\mathbf{A}+\mathbf{E}$ are both $n$-by-$n$ symmetric matrices, then for all $k\in[n]$,

[TABLE]

IV-B Main Differences in Proof Strategy vis-à-vis Spherical GMMs

For non-spherical GMMs or more general mixtures of log-concave distributions, we cannot expect a result similar to that mentioned in Remarks 6 and 7. That is, in general, the subspace spanned by the top $K$ singular vectors of $\mathbf{\Sigma}_{N}$ does not converge to the subspace spanned by the $K$ component mean vectors, where “convergence” is in the sense that the distance defined in (56) vanishes as $N\to\infty$. An illustration of this fact is presented in Kannan et al. [30, Figure 1]. However, we can still provide an upper bound for the distance of these two subspaces for more general mixture models. It is proved in Kannan et al. [30] that the subspace spanned by the top $K$ singular vectors of $\mathbf{\Sigma}_{N}$ is close (in terms of sample variances) to the means of samples generated from each component of the mixture (see Lemma 12 to follow). Define the maximum variance of a set of sample points generated from the $k$-th component of the mixture along any direction in a subspace $\mathcal{S}$ as

[TABLE]

where $\mathbf{c}_{k}:=\frac{1}{|\mathscr{C}_{k}|}\sum_{n\in\mathscr{C}_{k}}\mathbf{v}_{n}$ is the centroid of the points in $\mathscr{C}_{k}$. Recall that from Section I-B, we denote $\bar{\mathbf{v}}:=\frac{1}{N}\sum_{n=1}^{N}\mathbf{v}_{n}$ and $\mathbf{Z}$ as the centralized matrix of $\mathbf{V}$. Let $\bar{\mathbf{c}}_{k}=\mathbf{c}_{k}-\bar{\mathbf{v}}$. We have $\sigma_{k,\mathcal{S}}^{2}(\mathbf{Z})=\sigma_{k,\mathcal{S}}^{2}(\mathbf{V})$ for any subspace $\mathcal{S}$ and the the following lemma which is similar to Theorem 1 in Kannan et al. [30] holds. Note that this lemma holds not only for mixtures of log-concave distributions, but also for any mixture model.

Lemma 12**.**

Let $\mathscr{C}$ be the correct target clustering corresponding to the mixture. Let $\mathcal{W}$ be the subspace spanned by the top $K-1$ left singular vectors of $\mathbf{Z}$. Then

[TABLE]

where $\mathrm{d}(\bar{\mathbf{c}}_{k},\mathcal{W})$ denotes the orthogonal distance of $\bar{\mathbf{c}}_{k}$ from subspace $\mathcal{W}$ for any $k\in[K]$.

In addition, recall the notations $\mathbf{P}_{K-1},\mathbf{P}_{-(K-1)}$, and $\mathbf{Q}_{K-1}$ (cf. (41)) from Section III-D. Let $\bar{\mathbf{u}}_{k}:=\mathbf{u}_{k}-\bar{\mathbf{u}}$ for $k\in[K]$. Since $\sum_{k=1}^{K}w_{k}\bar{\mathbf{u}}_{k}=0$, $\mathrm{rank}(\bar{\mathbf{\Sigma}}_{0})\leq K-1$. It is easy to see that $\mathrm{d}(\mathbf{x},\mathcal{W})^{2}=\|\mathbf{x}-\mathbf{P}_{K-1}\mathbf{P}_{K-1}^{T}\mathbf{x}\|_{2}^{2}$ for any vector $\mathbf{x}$ and by denoting $\sigma^{2}_{k,\max}$ as the maximal eigenvalue of $\mathbf{\Sigma}_{k}$ (the covariance matrix of the $k$-th component), we obtain the following corollary of Lemma 12.

Corollary 5**.**

If we further assume that the mixture is a mixture of log-concave distributions, then for any sufficiently small $\epsilon\in(0,1)$ and all $\eta>0$, if $N\geq C\frac{F^{2}K^{4}}{\epsilon^{2}}\log^{5}\left(\frac{FK^{3}}{\epsilon\eta}\right)$, with probability at least $1-\eta$,

[TABLE]

The proof of the above corollary is presented in Appendix -B. Because Corollary 5 only provides a guarantee for the closeness of centralized component mean vectors to the SVD subspace of centralized samples, we need to further extend it to show that the subspace spanned by all the centralized component mean vectors is close to the SVD subspace of centralized samples. This is described in the following lemma.

Lemma 13**.**

Let $\mathbf{V}\in\mathbb{R}^{F\times N}$ be generated from a mixture of $K$ log-concave distributions that satisfies the non-degeneracy condition (cf. Definition 1). Using the notations defined above (see, in particular, the definition of $\lambda_{\min}$ in Section III-A),

[TABLE]

Proof.

We have that

[TABLE]

where $\mathbf{A}:=\mathbf{P}_{-(K-1)}^{T}\mathbf{U}\mathbf{D}$, $\mathbf{U}:=[\bar{\mathbf{u}}_{1},\ldots,\bar{\mathbf{u}}_{K}]$, and $\mathbf{D}:=\mathrm{diag}(\sqrt{w_{1}},\ldots,\sqrt{w_{K}})$. Recall that we write the SVD of $\bar{\mathbf{\Sigma}}_{0}:=\sum_{k=1}^{K}w_{k}\bar{\mathbf{u}}_{k}\bar{\mathbf{u}}_{k}^{T}$ as $\bar{\mathbf{\Sigma}}_{0}=\mathbf{Q}_{K-1}\mathbf{E}_{K-1}\mathbf{Q}_{K-1}^{T}$. We have

[TABLE]

where the inequality is because the diagonal entries of $\mathbf{Q}_{K-1}^{T}\mathbf{P}_{-(K-1)}\mathbf{P}_{-(K-1)}^{T}\mathbf{Q}_{K-1}$ are nonnegative.

Let $\beta:=\sum_{k=1}^{K}w_{k}\mathrm{d}(\bar{\mathbf{u}}_{k},\mathcal{W})^{2}$. By combining the above inequalities, we have

[TABLE]

Since

[TABLE]

where $\mathbf{Q}_{-(K-1)}\in\mathbb{R}^{F\times(F-K+1)}$ is selected such that $[\mathbf{Q}_{K-1},\mathbf{Q}_{-(K-1)}]$ is orthogonal, we also have that

[TABLE]

Now we have

[TABLE]

concluding the proof of Lemma 13. ∎

Combining the results of Corollary 5 and Lemma 13, we obtain an upper bound of the distance between the two subspaces, and we can prove a result concerning optimal clusterings of the dimensionality-reduced dataset (via PCA) generated from a mixture of log-concave distributions. This parallels the procedures for the proof strategy of Theorem 2.

IV-C Description of the Theorems for Log-Concave Distributions

For any $k\in[K]$, let $\sigma^{2}_{k,\max}$ and $\sigma^{2}_{k,\min}$ be the maximal and minimal eigenvalues of $\mathbf{\Sigma}_{k}$ respectively, and define $\bar{\sigma}^{2}_{\max}:=\sum_{k=1}^{K}w_{k}\sigma^{2}_{k,\max}$ and $\bar{\sigma}^{2}_{\min}:=\sum_{k=1}^{K}w_{k}\sigma^{2}_{k,\min}$. Other notations are the same as that in previous theorems. Then in a similar manner as Theorem 1 for spherical GMMs, we have the following theorem for mixtures of log-concave distributions.

Theorem 3**.**

Suppose that all the columns of data matrix $\mathbf{V}\in\mathbb{R}^{F\times N}$ are independently generated from a mixture of $K$ log-concave distributions and $N>F>K$. Assume the mixture model satisfies the non-degeneracy condition. We further assume that

[TABLE]

Then for any sufficiently small $\epsilon\in(0,1)$ and any $t\geq 1$, if $N\geq C\frac{K^{2}F^{4}}{\epsilon^{2}}\log^{5}\left(\frac{K^{2}F^{3}}{\epsilon\eta}\right)$, where $C>0$ depends on the parameters of the mixture model, we have, with probability at least $1-\eta$,

[TABLE]

where $\mathscr{C}^{\mathrm{opt}}$ is an optimal $K$-clustering for $\mathbf{V}$.

The proof, which is presented in Appendix -C, is mostly similar to that for Theorem 1, except that we employ different concentration inequalities and slightly different bounding strategies (e.g., Lemma 11 is required).

Next, we demonstrate that under similar assumptions on the generating process of the samples (compared to those in Theorem 3), any optimal clustering for the post-PCA (cf. Section II-C) dataset is also close to the correct target clustering with high probability.

Theorem 4**.**

Define $\bar{L}:=\sum_{k=1}^{K}w_{k}\left(\|\mathbf{u}_{k}\|_{2}^{2}+\mathrm{tr}(\mathbf{\Sigma}_{k})\right)$. Let the dataset $\mathbf{V}\in\mathbb{R}^{F\times N}$ be generated under the same conditions given in Theorem 3 with the separability assumption in (90) being modified to

[TABLE]

where

[TABLE]

Let $\tilde{\mathbf{V}}\in\mathbb{R}^{F\times(K-1)}$ be the post-$(K-1)$-PCA dataset of $\mathbf{V}$. Then for any sufficiently small $\epsilon\in(0,1)$, if $N\geq C\frac{F^{2}K^{6}}{\epsilon^{2}}\log^{5}\left(\frac{F^{2}K^{4}}{\epsilon\eta}\right)$, where $C>0$ depends on the parameters of the mixture model, we have, with probability at least $1-\eta$,

[TABLE]

where $\mathscr{C}$ is the correct target clustering and $\tilde{\mathscr{C}}^{\mathrm{opt}}$ is an optimal $K$-clustering for $\tilde{\mathbf{V}}$.

By using the fact that $\mathbf{w}=(w_{1},w_{2},\ldots,w_{K})$ is a probability vector and the non-degeneracy condition, we have $\sum_{k=1}^{K}w_{k}\|\mathbf{u}_{k}\|_{2}^{2}>\|\bar{\mathbf{u}}\|_{2}^{2}$ and thus $b>0$. The proof of Theorem 4, which can be found in Appendix -D, is similar to that for Theorem 2, except that the estimate of $\|\mathbf{P}_{K-1}\mathbf{P}_{K-1}^{T}-\mathbf{Q}_{K-1}\mathbf{Q}_{K-1}^{T}\|_{\mathrm{F}}$ is obtained differently.

IV-D Discussion and Comparison to Theorems for Spherical GMMs

The separability assumption (90) for mixtures of log-concave distributions reduces to (16) for spherical GMMs because in this case, we have $\bar{\sigma}^{2}_{\max}=\bar{\sigma}^{2}_{\min}=\bar{\sigma}^{2}$. If the mixture model is non-spherical, the separability assumption (90) is generally stricter than (16). This is especially the case when $\bar{\sigma}^{2}_{\max}\gg\bar{\sigma}^{2}_{\min}$. This implies that non-spherical mixture models are generally more difficult to disambiguate and learn. For dimensionality-reduced datasets (using PCA), the separability assumption in (92) is stricter than the separability assumption in (24), even for spherical GMMs, because of the presence of the additional positive terms $a$ and $b$ in (93). In addition, unlike that for spherical GMMs, the separability assumption in (92) for dimensionality-reduced datasets may also be stricter than the separability assumption in (90) also because of the same additional positive terms $a$ and $b$.

V Other Extensions

In this section, we discuss several interesting and practically-relevant extensions of the preceding results. In Section V-A, we show that Theorems 2 and 4 may be readily extended to other dimensionality-reduction techniques besides PCA/SVD by leveraging results such as (12). In Section V-B, we show that we can apply efficient clustering algorithms to obtain an approximately-optimal clustering which is also close to the correct target clustering.

V-A Other Dimensionality-Reduction Techniques

Our results can be used to prove similar upper bounds for the ME distance between any approximately-optimal clustering and the correct target clustering. The following corollary follows easily from Lemma 1.

Corollary 6**.**

Consider a $K$-clustering $\mathscr{C}$ with corresponding $\delta$ (cf. Lemma 1) and a $K$-clustering $\mathscr{C}^{\prime}$ that satisfies

[TABLE]

for some $\gamma\geq 1$. Then if

[TABLE]

satisfies

[TABLE]

we have

[TABLE]

Proof.

We have $\delta\leq\delta_{\gamma}\leq\frac{1}{2}(K-1)$ and $\tau(\delta,\delta_{\gamma})\leq\tau(\delta_{\gamma})\leq p_{\min}$. Lemma 1 thus yields $\mathrm{d}_{\mathrm{ME}}(\mathscr{C},\mathscr{C}^{\prime})\leq p_{\max}\tau(\delta)$. ∎

According to the above corollary, we deduce that if we make a stronger separability assumption as in (97), we can bound the ME distance between any approximately-optimal clustering and the correct target clustering. Therefore, by leveraging (12), our theoretical results for dimensionality reduction by PCA (in Theorems 2 and 4) can be extended to other dimensionality-reduction techniques such as random projection [32, 33, 34, 35] and randomized SVD [36, 35, 37]. We describe these dimensionality-reduction techniques in the following and provide known results for $\gamma$ satisfying (12).

• •

A random projection from $F$ dimensions to $D<F$ dimensions is represented by a $D\times F$ matrix, which can be generated as follows [38]: (i) Set each entry of the matrix to be an i.i.d. $\mathcal{N}(0,1)$ random variable; (ii) Orthonormalize the rows of the matrix. Theoretical guarantees for dimensionality-reduction via random projection are usually established by appealing to the well-known Johnson-Lindenstrauss lemma [39] which says that pairwise distances and inner products are approximately preserved under the random projection.

• •

Because computing an exact SVD is generally expensive, randomized SVD has gained tremendous interest in solving large-scale problems. For a data matrix $\mathbf{V}\in\mathbb{R}^{F\times N}$, to reduce the dimensionality of the columns from $F$ to $K<F$, one performs a randomized SVD using an $F\times K$ matrix $\mathbf{Z}_{K}$. More specifically, we can adopt the following procedure [24]: (i) Generate a $D\times F$ ($D>K$) matrix $\mathbf{L}$ whose entries are i.i.d. $\mathcal{N}(0,1)$ random variables; (ii) Let $\mathbf{A}=\mathbf{L}\mathbf{V}\in\mathbb{R}^{D\times N}$ and orthonormalize the rows of $\mathbf{A}$ to construct a matrix $\mathbf{B}$; (iii) Let $\mathbf{Z}_{K}\in\mathbb{R}^{F\times K}$ be the matrix of top $K$ left singular vectors of $\mathbf{V}\mathbf{B}^{T}\in\mathbb{R}^{F\times D}$. Such $\mathbf{Z}_{K}$ is expected to satisfy $\|\mathbf{V}-\mathbf{Z}_{K}\mathbf{Z}_{K}^{T}\mathbf{V}\|_{\mathrm{F}}\approx\|\mathbf{V}-\mathbf{P}_{K}\mathbf{P}_{K}^{T}\mathbf{V}\|_{\mathrm{F}}$, where $\mathbf{P}_{K}$ is the matrix of top $K$ left singular vectors of $\mathbf{V}$. The key advantage of randomized SVD over exact SVD is that when $D\ll\min\{F,N\}$, the computation of the randomized SVD is significantly faster than the computing of an exact SVD.

• •

We may also employ feature selection techniques such as those described in Boutsidis et al. [40] and Boutsidis and Magdon-Ismail [41].

A subset of the results of Cohen et al. [25] is presented in Table I. This table shows the $\gamma$ such that (12) is satisfied for various reduced dimensions and dimensionality reduction techniques. Even though the results in Table I appear promising, we observe from numerical experiments that for dimensionality reduction by PCA/SVD, if the data matrix is generated from a spherical GMM, even if it is moderately-well separated (cf. Section III-B to follow), $\mathcal{D}(\mathbf{V},\tilde{\mathscr{C}}^{\mathrm{opt}})\approx\mathcal{D}(\mathbf{V},\mathscr{C}^{\mathrm{opt}})$. That is, in this case, $\gamma\approx 1$ even though the reduced dimensionality is $K-1$ or $K$. Furthermore, we show in Theorem 2 that for dimensionality reduction by PCA, we require a weaker separability assumption (compared to that in Theorem 1). However, from Table I, for SVD, if the reduced dimensionality is $K$, then $\gamma=2$ and we will require a stronger separability assumption according to Corollary 6. Therefore, the results for PCA/SVD in Table I are generally pessimistic. This is reasonable because PCA/SVD is data-dependent and we have assumed specific generative mixture models for our data matrices.

V-B Efficient Clustering Algorithms

Although $k$-means is a popular heuristic algorithm that attempts to minimize the sum-of-squares distortion, in general, minimizing this objective is NP-hard and $k$-means only converges to a locally optimal solution. In addition, $k$-means is sensitive to initialization [5]. Fortunately, there are variants of $k$-means with judiciously chosen initializations that possess theoretical guarantees, e.g., $k$-means++ [5]. In addition, efficient variants of $k$-means [43, 44] have been proposed to find approximately-optimal clusterings under appropriate conditions. Our theoretical results can be easily combined with these efficient algorithms to produce approximately-optimal clusterings which are also close to the correct target clustering. We demonstrate this by using a result in Ostrovsky et al. [43]. Namely, if we denote the optimal distortion with $k\in\mathbb{N}$ clusters as $\mathrm{OPT}_{k}$, Theorem 4.13 in Ostrovsky et al. [43] states that:

Lemma 14**.**

If $\frac{\mathrm{OPT}_{K}}{\mathrm{OPT}_{K-1}}\leq\epsilon^{2}$ for a small enough $\epsilon>0$, the randomized algorithm presented before Theorem 4.13 in [43] returns a solution of cost at most $\left(\frac{1-\epsilon^{2}}{1-37\epsilon^{2}}\right)\mathrm{OPT}_{K}$ with probability $1-O(\sqrt{\epsilon})$ in time $O(FNK+K^{3}F)$.

We demonstrate that this lemma and the proposed algorithm can be combined with our theoretical results to produce further interesting results. For simplicity, we assume the data matrix $\mathbf{V}\in\mathbb{R}^{F\times N}$ is generated from a $K$-component spherical GMM. Then by previous calculations, the lower bound of the distortion for $K-1$ clusters is $\mathcal{D}_{K-1}^{*}:=\sum_{k=K-1}^{F}\lambda_{k}(\mathbf{S})$ (cf. Section II-B). As $N\to\infty$, $\mathcal{D}_{K-1}^{*}$ converges to $\lambda_{\min}+(F-K+2)\bar{\sigma}^{2}$ in probability. In addition, the distortion for the correct target clustering (with $K$ clusters) converges to $F\bar{\sigma}^{2}$ in probability. Therefore, if the number of samples is large enough, with high probability,

[TABLE]

Thus if $\bar{\sigma}^{2}$ is sufficiently small or $\lambda_{\min}$ is sufficiently large, by Lemma 14, we can use the algorithm suggested therein to obtain an approximately-optimal clustering for the original dataset. In addition, by Theorem 1 and Corollary 6, under an appropriate separability assumption, the approximately-optimal clustering is close to the correct target clustering that we ultimately seek.

We have provided one example of an efficient algorithm to obtain an approximately-optimal clustering. Interested readers may refer to the paper by Ackerman and Ben-David [44] which discusses other computationally efficient algorithms with guarantees.

VI Conclusion and Future Work

This paper provides a fundamental information-theoretic understanding about when optimizing the objective function of the $k$-means algorithm returns a clustering that is close to the correct target clustering. To ameliorate computational and memory issues, various dimensionality reduction techniques such as PCA are also considered.

Several natural questions arise from the work herein.

Instead of the separability assumptions made herein, we may consider modifying our analyses so that we eventually make less restrictive pairwise separability assumptions. This may enable us to make more direct comparisons between our separability assumptions and similar assumptions in the literature, such as those in Vempala and Wang [10] and Kannan et al. [30]. 2. 2.

Brubaker [31] considers the robust learning of mixtures of log-concave distributions. Similarly, we may extend our work to the robust learning of noisy mixtures in which there may be outliers in the data. 3. 3.

Besides studying the sum-of-squares distortion measure for $k$-means in (1), it may be fruitful to analyze other objective functions such as those for $k$-medians [45] or min-sum clustering [46]. These may result in alternative separability assumptions and further insights on the fundamental limits of various clustering tasks. 4. 4.

We have provided upper bounds on the ME distance under certain sufficient (separability) conditions. It would be fruitful to also study necessary conditions on the separability of the mixture components to ensure that the ME distance is small. This will possibly result in new separability assumptions which will, in turn, aid in assessing the tightness of our bounds and how they may be improved.

-A Complete Proof of Theorem 1

Proof.

To apply Corollary 1, we first estimate $\frac{1}{N}\mathcal{D}(\mathbf{V},\mathscr{C})$. We have

[TABLE]

where $n_{k}:=|\mathscr{C}_{k}|$. By Lemma 5 and by the property that if $X$ has a sub-Gaussian distribution,333The sub-Gaussian norm of a sub-Gaussian random variable $X$ is defined as $\|X\|_{\Psi_{2}}:=\sup_{p\geq 1}p^{-1/2}\left(\mathbb{E}|X|^{p}\right)^{1/p}$. $\|X-\mathbb{E}X\|_{\Psi_{2}}\leq 2\|X\|_{\Psi_{2}}$ (similarly, if $X$ has a sub-Exponential distribution, $\|X-\mathbb{E}X\|_{\Psi_{1}}\leq 2\|X\|_{\Psi_{1}}$) [28], we have for $k\in[K]$,

[TABLE]

where $C_{0}>0$ is a constant depending on $w_{k}$, $k\in[K]$. Then with probability at least $1-Ke\exp(-C_{0}N)$, we have $\frac{n_{k}}{N}\geq\frac{w_{k}}{2}$ for all $k\in[K]$. For brevity, we only consider this case and replace $n_{k}$ with $N$ in the following inequalities. In addition, for any $0<\epsilon\leq\frac{w_{\min}}{2}<1$,

[TABLE]

where $\tau_{k}>0$ is a constant depending on $w_{k}$.

By Lemma 7, the square of each entry of a random vector generated from a spherical Gaussian distribution has a sub-Exponential distribution. In addition, for random vector $\mathbf{v}$ generated from the $k$-th component of the spherical GMM, we have $\mathbb{E}[\mathbf{v}(f)^{2}]=\mathbf{u}_{k}(f)^{2}+\sigma_{k}^{2}$ for any $f\in[F]$. By Lemma 6, for any $k\in[K]$,

[TABLE]

where $\xi_{k}>0$ is a constant depending on $w_{k},\sigma_{k}^{2},\mathbf{u}_{k}$. Similarly, by Lemma 5, there exists $\zeta_{k}>0$ depending on $w_{k}$, $\sigma_{k}^{2}$ and $\mathbf{u}_{k}$, such that

[TABLE]

The final bound in (-A.9) holds because for any $f$, if $\mathbf{u}_{k}(f)=0$, we have

[TABLE]

On the other hand, if $\mathbf{u}_{k}(f)\neq 0$, note that $\left|\mathbf{c}_{k}(f)^{2}-\mathbf{u}_{k}(f)^{2}\right|\geq\frac{\epsilon}{F}$ implies either $\left|\mathbf{c}_{k}(f)-\mathbf{u}_{k}(f)\right|\geq\frac{\epsilon}{3|\mathbf{u}_{k}(f)|F}$ or $\left|\mathbf{c}_{k}(f)+\mathbf{u}_{k}(f)\right|\geq 3|\mathbf{u}_{k}(f)|$. We have

[TABLE]

Now let $d_{k}:=\frac{1}{n_{k}}\sum_{n\in\mathscr{C}_{k}}\|\mathbf{v}_{n}\|_{2}^{2}-\|\mathbf{c}_{k}\|_{2}^{2}$ and $L_{k}:=\|\mathbf{u}_{k}\|_{2}^{2}+F\sigma_{k}^{2}$. Then, there exists a constant $C_{1}>0$ depending on $\{(w_{k},\sigma_{k}^{2},\mathbf{u}_{k})\}_{k\in[K]}$ such that

[TABLE]

where (-A.14) is a consequence of (-A.3). This proves (36).

Now we estimate the positive eigenvalues of $\mathbf{S}:=\mathbf{Z}^{T}\mathbf{Z}$, where $\mathbf{Z}$ is the centralized data matrix of $\mathbf{V}$. Equivalently, we may consider the eigenvalues of $\bar{\mathbf{\Sigma}}_{N}:=\frac{1}{N}\mathbf{Z}\mathbf{Z}^{T}=\frac{1}{N}\sum_{n=1}^{N}(\mathbf{v}_{n}-\bar{\mathbf{v}})(\mathbf{v}_{n}-\bar{\mathbf{v}})^{T}$, where $\bar{\mathbf{v}}=\frac{1}{N}\sum_{n}\mathbf{v}_{n}$. The expectation of centralized covariance matrix for the spherical GMM is $\sum_{k=1}^{K}w_{k}(\mathbf{u}_{k}-\bar{\mathbf{u}})(\mathbf{u}_{k}-\bar{\mathbf{u}})^{T}+\bar{\sigma}^{2}\mathbf{I}=\bar{\mathbf{\Sigma}}.$ For any $f\in[F]$,

[TABLE]

and

[TABLE]

Hence, similarly to (-A.9), and by (-A.19), we obtain

[TABLE]

where $C_{3}>0$ is a constant depending on $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$. Note that

[TABLE]

Then similar to (-A.16) and by Lemma 8, we have that for any $t\geq 1$, if $N\geq C_{4}F^{3}K^{2}t^{2}/\epsilon^{2}$, where $C_{4}>0$ is a constant depending on $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$,

[TABLE]

This proves (38).

Now, if $N\geq C_{4}F^{5}K^{2}t^{2}/\epsilon^{2}$, we have

[TABLE]

This proves (37).

In addition, if $N\geq C_{4}F^{3}K^{2}t^{2}/\epsilon^{2}$, similarly,

[TABLE]

and

[TABLE]

This proves (39) and (40).

Finally, let $p_{\min}=\frac{1}{N}\min_{k}|\mathscr{C}_{k}|$ and $p_{\max}=\frac{1}{N}\max_{k}|\mathscr{C}_{k}|$. Then if $\epsilon>0$ satisfies (17), when $N\geq CF^{5}K^{2}t^{2}/\epsilon^{2}$ with $C>0$ being a constant depending on $\{(w_{k},\sigma_{k}^{2},\mathbf{u}_{k})\}_{k\in[K]}$, with probability at least $1-36F^{2}K\exp\left(-t^{2}F\right)$,

[TABLE]

Note that both $\tau(\mathord{\cdot})$ and $\zeta(\mathord{\cdot})$ are continuous and monotonically increasing functions on $[0,\frac{1}{2}(K-1)]$. Thus by the separability condition (17), we have

[TABLE]

where the second inequality is from (-A.5). Note that $\tau(\zeta(p_{\min}))=p_{\min}$. Then (-A.37) is equivalent to

[TABLE]

Therefore, by Corollary 1, if $N\geq CF^{5}K^{2}t^{2}/\epsilon^{2}$ (where $C>0$ is an appropriate constant depending on $\{(w_{k},\mathbf{u}_{k},\sigma_{k}^{2})\}_{k\in[K]}$),

[TABLE]

with probability at least $1-36KF^{2}\exp\left(-t^{2}F\right)$. ∎

-B Proof of Corollary 5

Proof.

Let $\mathbf{p}_{\bar{\mathbf{c}},k}=\mathbf{P}_{-(K-1)}\mathbf{P}_{-(K-1)}^{T}\bar{\mathbf{c}}_{k}$ and $\mathbf{p}_{\bar{\mathbf{u}},k}=\mathbf{P}_{-(K-1)}\mathbf{P}_{-(K-1)}^{T}\bar{\mathbf{u}}_{k}$. Consider,

[TABLE]

where (-B.2) is because that $\mathbf{P}_{-(K-1)}\mathbf{P}_{-(K-1)}^{T}$ is a projection matrix and $C_{1}>0$ is a constant depending on parameters of the mixture model. On the other hand,

[TABLE]

where $\lambda_{1}(\cdot)$ denotes the largest eigenvalue of a matrix. Consequently, for some fixed $\eta^{\prime}\in(0,1)$, if $N\geq C_{2}\frac{F^{2}K^{4}}{\epsilon^{2}}\log^{5}\left(\frac{FK^{2}}{\epsilon\eta^{\prime}}\right)$ (where $C_{2}>0$ is a constant depending on parameters of the mixture model), we have

[TABLE]

where (-B.6) is due to Lemma 2 and (-B.7) is due to the fact that

[TABLE]

Then with probability at least $1-(8e+2)K\eta^{\prime}$,

[TABLE]

Now, we define $\eta:=(8e+2)K\eta^{\prime}$. Then, we obtain (75) with the sample complexity result as in the statement of Corollary 5. ∎

-C Proof of Theorem 3

Proof.

We have that the two inequalities concerning $w_{k}$ in (-A.4) and (-A.5) still hold. In addition, for any $k\in[K]$, if $N\geq C_{1}\frac{F^{2}}{\epsilon^{2}}\log^{5}\left(\frac{F}{\epsilon\eta}\right)$ with $C_{1}>0$ being sufficiently large,

[TABLE]

By Lemma 6, similarly, we have for a sufficiently small $c>0$,

[TABLE]

Therefore, by $\sum_{k=1}^{K}w_{k}\mathrm{tr}(\mathbf{\Sigma}_{k})\leq F\bar{\sigma}^{2}_{\max}$, similar to that for spherical GMMs, if $N\geq C_{1}\frac{F^{2}K^{2}}{\epsilon^{2}}\log^{5}\left(\frac{F^{2}K^{2}}{\epsilon\eta}\right)$,

[TABLE]

Similarly, we have

[TABLE]

We also have that

[TABLE]

Note that the following bound for $\bar{\mathbf{\Sigma}}_{N}-\bar{\mathbf{\Sigma}}$ is slightly different from (-A.26) because Lemma 10 requires that the log-concave distribution has zero mean. Furthermore, recall that $\mathbf{c}_{k}:=\frac{1}{|\mathscr{C}_{k}|}\sum_{n\in\mathscr{C}_{k}}\mathbf{v}_{n}$, we have

[TABLE]

Therefore, we have that if $N\geq C_{1}\frac{F^{2}K^{2}}{\epsilon^{2}}\log^{5}\left(\frac{FK^{2}}{\epsilon\eta}\right)$, with probability at least $1-\eta$,

[TABLE]

Now, note that by Lemma 11, for $k\leq F<N$,

[TABLE]

Therefore, if $N\geq C_{1}\frac{F^{4}K^{2}}{\epsilon^{2}}\log^{5}\left(\frac{F^{2}K^{2}}{\epsilon\eta}\right)$, we have

[TABLE]

Or more concisely, if $N\geq C_{1}\frac{F^{4}K^{2}}{\epsilon^{2}}\log^{5}\left(\frac{F^{3}K^{2}}{\epsilon\eta}\right)$,

[TABLE]

Similarly, by the inequalities $\lambda_{K-1}(\bar{\mathbf{\Sigma}})\geq\lambda_{K-1}(\bar{\mathbf{\Sigma}}_{0})+\bar{\sigma}^{2}_{\min}$ and $\lambda_{K}(\bar{\mathbf{\Sigma}})\leq\bar{\sigma}^{2}_{\max}$, if $N\geq C_{1}\frac{F^{2}K^{2}}{\epsilon^{2}}\log^{5}\left(\frac{FK^{2}}{\epsilon\eta}\right)$,

[TABLE]

Combining these results with Corollary 1, we similarly obtain the conclusion we desire. ∎

-D Proof of Theorem 4

Proof.

We use the same notations as those in the proof of Theorem 2 and in the statement of Theorem 4. Since $\mathbf{Q}_{K-1}\in\mathbb{R}^{F\times(K-1)}$ has orthonormal columns,

[TABLE]

for all $k\in[K]$. Therefore, similar to that for the case for the original dataset (cf. the inequality in (-C.4)), if $N\geq C_{1}\frac{K^{4}}{\epsilon^{2}}\log^{5}\left(\frac{K^{4}}{\epsilon\eta}\right)$ with $C_{1}>0$ being sufficiently large,

[TABLE]

In addition, we have $\lambda_{K-1}(\mathbf{Q}_{K-1}^{T}\mathbf{\Sigma}_{k}\mathbf{Q}_{K-1})\geq\lambda_{F}(\mathbf{\Sigma}_{k})=\sigma_{k,\min}^{2}$.444Indeed, assume, to the contrary, that $\lambda_{K-1}(\mathbf{Q}_{K-1}^{T}\mathbf{\Sigma}_{k}\mathbf{Q}_{K-1})<\sigma_{k,\min}^{2}$. Then there is a $\lambda<\sigma_{k,\min}^{2}$ and a corresponding unit vector $\mathbf{x}\in\mathbb{R}^{K-1}$, such that $\mathbf{Q}_{K-1}^{T}\mathbf{\Sigma}_{k}\mathbf{Q}_{K-1}\mathbf{x}=\lambda\mathbf{x}$. Thus, $\sigma_{k,\min}^{2}\|\mathbf{x}\|_{2}^{2}=\sigma_{k,\min}^{2}\|\mathbf{Q}_{K-1}\mathbf{x}\|_{2}^{2}\leq\mathbf{x}^{T}\mathbf{Q}_{K-1}^{T}\mathbf{\Sigma}_{k}\mathbf{Q}_{K-1}\mathbf{x}=\lambda\|\mathbf{x}\|_{2}^{2}<\sigma_{k,\min}^{2}\|\mathbf{x}\|_{2}^{2}$, which is a contradiction. Similarly, we have $\lambda_{1}(\mathbf{Q}_{K-1}^{T}\mathbf{\Sigma}_{k}\mathbf{Q}_{K-1})\leq\lambda_{1}(\mathbf{\Sigma}_{k})=\sigma_{k,\max}^{2}$. Thus if $N\geq C_{1}\frac{K^{4}}{\epsilon^{2}}\log^{5}\left(\frac{K^{3}}{\epsilon\eta}\right)$,

[TABLE]

Recall that we write $\mathbf{R}:=\mathbf{Q}_{K-1}\mathbf{Q}_{K-1}^{T}-\mathbf{P}_{K-1}\mathbf{P}_{K-1}^{T}$. Let $r:=\sqrt{\frac{2(K-1)\bar{\sigma}_{\max}^{2}}{\lambda_{\min}}}$. Combining Corollary 5 and Lemma 13, we have that if $N\geq C\frac{F^{2}K^{4}}{\epsilon^{2}}\log^{5}\left(\frac{FK^{3}}{\epsilon\eta}\right)$, with probability at least $1-\eta$,

[TABLE]

where $C_{2}>0$ is sufficiently large. In addition, using the inequalities

[TABLE]

we deduce that

[TABLE]

Therefore, by (-C.2) and (-D.4), we obtain that if $N\geq C_{1}\frac{F^{2}K^{6}}{\epsilon^{2}}\log^{5}\left(\frac{F^{2}K^{4}}{\epsilon\eta}\right)$,

[TABLE]

Similarly, when $N\geq C_{1}\frac{F^{2}K^{6}}{\epsilon^{2}}\log^{5}\left(\frac{F^{2}K^{4}}{\epsilon\eta}\right)$,

[TABLE]

Combining these results with Corollary 1, we obtain the desired conclusion. ∎