Finding a latent k-simplex in O(k . nnz(data)) time via Subset Smoothing

TL;DR

This paper introduces an efficient geometric algorithm for latent variable models that finds a latent k-simplex in data with near-linear time complexity, improving scalability and robustness in various models.

Contribution

It proposes a novel subset smoothing technique and an algorithm that efficiently finds a latent k-polytope, unifying multiple models under a geometric framework with improved computational performance.

Findings

Runs in O*(k * nnz) time, matching or surpassing existing algorithms.

First quasi-input-sparsity time algorithm for certain models with small k.

Robustly estimates cluster centers even under adversarial data perturbations.

Abstract

In this paper we show that a large class of Latent variable models, such as Mixed Membership Stochastic Block(MMSB) Models, Topic Models, and Adversarial Clustering, can be unified through a geometric perspective, replacing model specific assumptions and algorithms for individual models. The geometric perspective leads to the formulation: \emph{find a latent $k-$ polytope $K$ in ${\bf R}^d$ given $n$ data points, each obtained by perturbing a latent point in $K$}. This problem does not seem to have been considered in the literature. The most important contribution of this paper is to show that the latent $k-$polytope problem admits an efficient algorithm under deterministic assumptions which naturally hold in Latent variable models considered in this paper. ur algorithm runs in time $O^*(k\; \mbox{nnz})$ matching the best running time of algorithms in special cases considered here and is better when the data is sparse, as is the case in applications. An important novelty of the algorithm is the introduction of \emph{subset smoothed polytope}, $K'$, the convex hull of the ${n\choose \delta n}$ points obtained by averaging all $\delta n$ subsets of the data points, for a given $\delta \in (0,1)$. We show that $K$ and $K'$ are close in Hausdroff distance. Among the consequences of our algorithm are the following: (a) MMSB Models and Topic Models: the first quasi-input-sparsity time algorithm for parameter estimation for $k \in O^*(1)$, (b) Adversarial Clustering: In $k-$means, if, an adversary is allowed to move many data points from each cluster an arbitrary amount towards the convex hull of the centers of other clusters, our algorithm still estimates cluster centers well.

Equations

$$A_{i,j}=\frac{1}{m}\sum _{t=1}^m{X}_{ijt},X_{ijt}\sim Bernoulli(P_{ij})$$

$$E(A_{ij})=P_{ij};Var(A_{ij}|P_{ij})=\frac{1}{m}{P}_{ij}(1-P_{ij})$$

$$E(|A_{\cdot ,j}-P_{\cdot ,j}{|}^2)=\frac{1}{m}(1-\sum _{i=1}^d{P}_{ij}^2)$$

$$\ge \frac{1}{m}(1-\text{mbox}{Max}_{\ell }\sum _{i=1}^d{M}_{i\ell }^2),$$

$$\sum _{i=1}^d{M}_{i\ell }^2\le \text{mbox}{Max}_{i,\ell }{M}_{i,\ell }=\gamma ,\text{mbox}say.$$

$$\varphi (W_{\cdot ,j})\propto \prod _{\ell =1}^k{W}_{\ell ,j}^{\beta -1},$$

$$\forall \ell ,\forall \zeta \text{Prob}\left(W_{\ell ,j}\ge 1-\zeta \right)\ge \frac{1}{3k}{\zeta }^2.$$

$$\mathbf{P}={\mathbf{W}}^T\mathbf{B}\mathbf{W}.$$

$$\mathbf{P}=\underset{\mathbf{M}}{\underbrace{({\mathbf{W}}^{(1)}{)}^T\mathbf{B}}}{\mathbf{W}}^{(2)}.$$

$$\sigma =\text{mbox}{Max}_{v:|v|=1}\sqrt{\frac{1}{n}|v^T(\mathbf{A}-\mathbf{P}){|}^2}=\frac{1}{\sqrt{n}}||\mathbf{A}-\mathbf{P}||;\frac{1}{n}\text{mbox}{Min}_{\ell }|C_{\ell }|\ge \delta ,$$

$$\forall j,P_{\cdot ,j}\in \text{Convex Hull of }(M_{\cdot ,1},M_{\cdot ,2},\dots ,M_{\cdot ,k})$$

$$\forall \ell \in [k],\exists S_{\ell },|S_{\ell }|=\delta n:\forall j\in S_{\ell },|P_{\cdot ,j}-M_{\cdot ,\ell }|\le 4\sigma /\sqrt{\delta }$$

$$A_{\cdot ,j},j=1,2,\dots ,n\text{mbox}mutuallyindependent|\mathbf{P}$$

$$A_{\cdot ,j}\text{mbox}aredrawanaccordingtoaspecificprobdistributionsatisfying$$

$$E(A_{\cdot ,j}|P_{\cdot ,j})=P_{\cdot ,j}\text{mbox}and$$

$$\text{mbox}Anupperboundon||\text{mbox}Var(A_{\cdot ,j}|P_{\cdot ,j})||.$$

$$\frac{1}{\sqrt{n}}|||\mathbf{A}-\mathbf{P}||=\sigma$$

$$\mathbf{P}=\mathbf{M}\mathbf{W}.$$

$$\forall \ell \in [k],\text{Prob}\left(|P_{\cdot ,j}-M_{\cdot ,\ell }|\le {\epsilon }_1\right)\ge {\delta }_1,$$

$$A_{\cdot ,R}=\frac{1}{|R|}\sum _{j\in R}{A}_{\cdot ,j};P_{\cdot ,R}=\frac{1}{|R|}\sum _{j\in R}{P}_{\cdot ,j}.$$

$$\text{Dist}(K_1,K_2)=\underset{x\in K_1}{\sup }\underset{y\in K_2}{\inf }|x-y|.$$

$$\frac{\sigma }{\sqrt{\delta }}\le \frac{\text{mbox}{Min}_{\ell }|M_{\cdot ,\ell }|}{\text{poly}(k)}.$$

$$\forall \ell \in [k],\left|\text{mbox}Proj\left(M_{\cdot ,\ell },\text{mbox}Null\left(\mathbf{M}\setminus M_{\cdot ,\ell }\right)\right)\right|\ge \alpha \text{mbox}{Max}_{{\ell }^{\prime }}|M_{\cdot ,{\ell }^{\prime }}|.$$

$$\text{For}\ell \in [k],\exists S_{\ell }\subseteq [n],|S_{\ell }|=\delta n,\text{ with }|M_{\cdot ,\ell }-P_{\cdot ,j}|\le \frac{4\sigma }{\sqrt{\delta }}\forall j\in S_{\ell }.$$

$$\frac{\sigma }{\sqrt{\delta }}\le \frac{{\alpha }^3\text{mbox}{Min}_{\ell }|M_{\cdot ,\ell }|}{4500k^9}.$$

$$|A_{\cdot ,R_{\ell }}-M_{\cdot ,\ell }|\le \frac{150k^4}{\alpha }\frac{\sigma }{\sqrt{\delta }}\text{for}\ell =1,2,\dots ,k.$$

$$|A_{\cdot ,R_t}-M_{\cdot ,{\ell }_t}|\le \frac{150k^4}{\alpha }\frac{\sigma }{\sqrt{\delta }}\text{mbox}fort=1,2,\dots ,r.$$

$$U=V\cap \text{mbox}Null(A_{\cdot ,R_1},A_{\cdot ,R_2},\dots ,A_{\cdot ,R_r})$$

$$S=\arg \underset{T\subseteq [n],|T|=\delta n}{\max }|u\cdot A_{\cdot ,T}|.$$

$$\exists \ell \notin \{{\ell }_1,{\ell }_2,\dots ,{\ell }_r\}\text{mbox}suchthat|M_{\cdot ,\ell }-A_{\cdot ,S}|\le \frac{150k^4}{\alpha }\frac{\sigma }{\sqrt{\delta }}.$$

$$|M_{\cdot ,\ell }|\in \Omega (1).$$

$$|{\tilde{M}}_{\cdot ,\ell }-M_{\cdot ,\ell }|\le \frac{c{k}^{4.5}}{\alpha m^{1/4}}\text{for}\ell =1,2,\dots ,k.$$

$$\nu =\text{mbox}Max(\text{mbox}{Max}_i\sum _j{P}_{ij},\text{mbox}{Max}_j\sum _i{P}_{ij}).$$

Full text

We first observe that for a suitable $r\leq n$, $K$ is close to a data-determined polytope $K^{\prime}$ (the subset smoothed polytope) which is the convex hull of the ${n\choose r}$ points, each obtained by averaging an $r$ subset of data points. Our algorithm is simply stated: it optimizes $k$ carefully chosen linear functions over $K^{\prime}$ to find the $k$ vertices of the latent simplex. The proof of correctness is more involved, drawing on existing and new tools from Numerical Analysis. Our running time is $O^{*}(k\mbox{ nnz})$ (This is the time taken by one itertion of the $k-$means algorithm.) This is better than all previous algorithms for the special cases when data is sparse, as is the norm for Topic Modeling and MMBM. Some consequences of our algorithm are:

• •

Mixed Membership Models and Topic Models: We give the first quasi-input-sparsity time algorithm for $k\in O^{*}(1)$.

• •

Adversarial Clustering: In $k-$means, an adversary is allowed to move many data points from each cluster towards the convex hull of other cluster centers. Our algorithm still estimates cluster centers well.

1 Introduction

Understanding the underlying generative process of observed data is an important goal of Unsupervised learning. The setup is assumed to be stochastic where each observation is generated from a probability distribution parameterized by a model. Discovering such models from observations is a challenging problem, often intractable in the general. $k-$means Clustering of data is a simple and important special case. It is often used on data which is assumed to be generated by a mixture model like Gaussian Mixtures, where, all observations are generated from a fixed convex combination of density functions. The $k$-means problem, despite its simplicity, poses challenges and continues to attract considerable research attention. Mixed membership models [2] are interesting generalizations of Mixture models, where instead of a fixed convex combination, each observation arises from a different convex combination, often determined stochastically. Special cases of Mixed membership models include Topic Models [10], and Mixed Membership Stochastic Block(MMSB) models [1] which have gained significant attention for their ability to model real-life data.

The success of discovered models, though approximate, in explaining real-world data has spurred interest in deriving algorithms with proven upper bounds on error and time which can recover the true model from a finite sample of observations. This line of work has seen growing interest in algorithms community  [6, 3, 14, 8] . We note that these papers make domain-specific assumptions under which the algorithms are analyzed.

Mixture Models, Topic Models, and MMSB are all instances of Latent Variable models. Our aim in this paper is to arrive at a simply stated, general algorithm which is applicable to large class of Latent Variable models and which in polynomial time can recover the true model if certain general assumptions are satisfied. We take a geometric perspective on latent variable models and argue that such a perspective can serve as a unifying view yielding a single algorithm which is competitive with the state of the art and indeed better for sparse data.

2 Latent Variable models and Latent k-simplex problem

This section reviews three well-known Latent variable models: Topic models (LDA), MMSB and Clustering. The purpose of the review is to bring out the fact that all of them can be viewed abstractly as special cases of a geometric formulation which we call the Learning a Latent Simplex (LLS) problem.

Given (highly)pertubed points from a $k-$simplex in ${\bf R}^{d}$, learn the $k$ vertices of $K$.

In total generliaty, this problem is intractable. A second purpose of the review of LDA and MMSB is to distill out the domain-specific model assumptions they make into general determinsitic geometric assumptions on LLS. With these deterministic assumptions in place, our main contribution of the paper is to devise a fast algorithm to solve LLS.

2.1 Topic models and LDA

Topic Models attempt to capture underlying themes of a document by topics, which are probability distributions over all words in the vocabulary. Given a corpus of documents, each document is represented by relative word frequencies. Assuming that the corpus is generated from an ad-mixture of these $k-$distributions or $k-$topics, the core goal of Topic models is to construct the underlying $k$ topics. These can also be viewed geometrically as $k$ latent vectors $M_{\cdot,1},M_{\cdot,2},\ldots,M_{\cdot,k}\in{\bf R}^{d}$ ($d$ is the size of the vocabulary) where, $M_{i,\ell}$ is the expected frequency of word $i$ in topic $\ell$. Each $M_{\cdot,\ell}$ has non-negative entries summing to 1. In Latent Dirichlet Allocation (LDA) ([9]), an important example of Topic models, the data generation is stochastic. A document consisting of $m$ words is generated by the following two stage process:

• •

The topic distribution of Document $j$ is decided by the topic weights, $W_{\ell,j},\ell=1,2,\ldots,k$ picked from a Dirchlet distribution on the unit simplex $\{x\in{\bf R}^{k}:x_{\ell}\geq 0;\sum_{\ell=1}^{k}x_{\ell}=1\}$. The topic of the document $j$ is set to $P_{\cdot,j}=\sum_{\ell=1}^{k}M_{\cdot,\ell}W_{\ell,j}$. $P_{j}$ are latent points.

• •

The $m$ words of document $j$ are generated in i.i.d. trials from the multinomial distribution with $P_{\cdot,j}$ as the probability vector. The data point $A_{\cdot,j}$ is given by:

[TABLE]

The random-variate, $X_{ijt}=1$ if $i$ th word was chosen in the $t$ th draw while generating the $j$ th document and [math] otherwise. In other words $A_{ij}$ is the relative frequency of $i$th word in the $j$th document. As a consequence,

[TABLE]

The data generation process of a Topic model, such as LDA, can be also viewed from a geometric perspective. For each document, $j$, the observed data, $A_{\cdot,j}$, is generated from $P_{\cdot,j}$, a point in the simplex, $K$, whose vertices are defined by the $k-$ Topic vectors.

If priors are sepcified then recovering ${\bf M}$ in Topic Modeling, such as LDA, can be viewed as a classical parameter estimation problem, where, one is given samples (here $A_{\cdot,j}$) drawn from a multi-variate probability distribution and the problem is to estimate the parameters of the distribution. The learning problem in LDA is usually addressed by two classes of algorithms- Variational and MCMC based, neither of which is known to be polynomial time bounded. Recently, polynomial time algorithms have been developed for Topic Modeling under assumptions on word frequencies, topic weights and Numerical Analysis properties (like condition number) of the matrix ${\bf M}$[6, 3, 8]. While the algorithms are provably polynomial time bounded, the assumptions are domain-specific and the running time is not good as the algorithm we present here. Also, it is to be noted that the algorithm to be presented here is completely different in approach from the ones in the literature.

Topic modeling can be posed geometrically as the problem of learning the latent k-simplex, namely, the convex hull of the topic vectors. To formalize the problem and devcise an algorithm it will be useful to understand some properties of the data generated from such simplices, which we do presently. For concretenss, we focus on LDA, a widely used Topic model.

Data Outside $K$: The convex hull of the topic vectors $M_{\cdot,1},M_{\cdot,2},\ldots,M_{\cdot,k}$ we assume is a simplex (namely, they span a $k$ dimensional space). It is denoted $K$. $K$ is in ${\bf R}^{d}$ and so $d\geq k$; indeed, generally, $d>>k$.

We point out here that data points $A_{\cdot,j}$ can lie outside $K$. Indeed, even in the case when $k=1$, whence, $K=\{M_{\cdot,1}\}$, $A_{\cdot,j}$ will lie outside $K$ since: in the usual parameter setting, $m,k\in O^{*}(1)$, and, $n$ goes to infinity and the tail of $M_{\cdot,\ell}$, namely, $I=\{i:M_{i,\ell}\leq 1/(2m)\}$ has $\sum_{i\in I}M_{i,\ell}\in\Omega(1)$. So, for most $j$, there is at least one $i\in I$ with $A_{ij}\geq 1/m$ which implies $A_{\cdot,j}\not=M_{\cdot,1}$. While for $k=1$, there are simpler examples, we have chosen this illustration because it is easy to extend the argument for general $k$.

Not only does data often lie outside $K$, we argue below that in fact it lies significantly outside. This property of data being outside $K$ distinguishes our problem from the extensive literature in Theoretical Computer Science (see e.g. [5]) on learning polytopes given (often uniform random) sample inside the polytope. A simple calculation, which easily follows from (2.1), shows that

[TABLE]

where the last inequality follows by noting that $P_{\cdot,j}$ is a convex combination of columns of ${\bf M}$ and $x^{2}$ is a convex function of $x$.

[TABLE]

Now, $\gamma$ the maximum frequency of a single word in a topic, which is usually at most a small fraction. So, individual $|A_{\cdot,j}-P_{\cdot,j}|$ which we refer to as the “perturbation” of point $j$, is $\Omega(1/\sqrt{m})$, which is $\Omega^{*}(1)$ in the usual parameter ranges. But note that a side of $K$, namely $|M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}|\leq 1$, so perturbations can be the same order as sides of $K$. To summarize in words: most/all data points can lie $\Omega(\mbox{ side length of }K)$ outside of $K$.

Subset Averages While individual $|A_{\cdot,j}-P_{\cdot,j}|$ may be high, intuitively, the average of $A_{\cdot,j}$ over a large subset $R$ of $[n]$ should be close to the average of $P_{\cdot,j}$ over $R$ by law of large numbers. Indeed, we will prove later an upper bound on the spectral norm of the matrix of pereturbations ${\bf A}-{\bf P}$ (see Lemma 7.1 gives a precise statement) by using the stochastic independence of words in documents and applying Random Matrix Theory. This upper bound immediately implies that simultaneously for every $R\subseteq[n]$, we have a good upper bound on $|A_{\cdot,R}-P_{\cdot,R}|$ as we will see in Lemma (3.1). This leads us to our starting point for an algorithmic solution, namely, a technique we will call Subset Smoothing. Subset Smoothing is the obserbvation that if we take the convex hull $K^{\prime}$ of the ${n\choose\delta n}$ averages of all $\delta n$-sized subsets of the $n$ data points, then, $K^{\prime}$ provisdes a good approximation to $K$ under our assumptions. (Theorem  3.1 states precisely how close $K,K^{\prime}$ are.) We next describe another property of LDA which is essential for subset smoothing.

Proximity Recall that LDA posits a Dirichlet prior on picking topic weights $W_{\ell,j}$. The Dirichlet prior $\phi(\cdot)$ on the unit simplex is given by

[TABLE]

where, $\beta$ is a parameter, often set of a small positive value like $1/k$. Since $\beta<1$, this density puts appreciable mass near the corners. Indeed, one can show (see Lemma (9.3)) that

[TABLE]

So, a good fraction of the $P_{\cdot,j}$ are near each corner of $K$. This indeed helps the learning algorithm, since, for example, if all $P_{\cdot,j}$ lay in a proper subset of $K$, it is difficult to learn $K$ in general. Quantitatively, we will see that this leads a lower bound on the fraction of $j$ with $P_{\cdot,j}\approx M_{\cdot,\ell}$.

Input Sparsity Each data point $A_{\cdot,j}$ has at most $m$ non-zero entries and as we stated earlier, typcally $m<<d,n$ and so the data is sparse. It is important to design algorithms which exploit this sparsity.

2.2 Mixed Membership Stochastic Block(MMSB) Models

Formulated in [1], this is a model of a random graph where edge $(j_{1},j_{2})$ is in the graph iff person $j_{1}$ knows person $j_{2}$. There are $k$ communities; there is a $k\times k$ latent matrix ${\bf B}$, where, $B_{\ell_{1},\ell_{2}}$ is the probability that a person in community $\ell_{1}$ knows a person in community $\ell_{2}$. An underlying stochastic process is posited, again consisting of two components: For $j=1,2,\ldots,n$, person $j$ picks a $k$ vector $W_{\cdot,j}$ of community membership weights (non-negative reals summing to 1) according to a Dirichlet probbaility density. This is akin to the prior on picking ${\bf W}$ in LDA. Then, the edges are picked independently. For edge $(j_{1},j_{2})$, person $j_{1}$ picks a community $\ell$ from the multinomial with probabilities given by $W_{\cdot,j_{1}}$ and person $j_{2}$ picks a community $\ell^{\prime}$ according to $W_{\cdot,j_{2}}$. Then, edge $(j_{1}.j_{2})$ is included in $G$ with probability $B_{\ell,\ell^{\prime}}$. So, it is easy to see that the probaility $P_{j_{1},j_{2}}$ of edge $(j_{1},j_{2})$ being in $G$ is given by $P_{j_{1},j_{2}}=\sum_{\ell,\ell^{\prime}}W_{\ell,j_{1}}B_{\ell,\ell^{\prime}}W_{\ell^{\prime},j_{2}}$ which in matrix notation reads:

[TABLE]

${\bf W}$ being a stochastic matrix, it cannot be recovered from $G$, but we can aim to recover ${\bf B}$. But now ${\bf P}$ depends quadratically on ${\bf W}$ and recovering ${\bf B}$ directly does not seem easy. Indeed, the only provable polynomial time algorithms known to date for this problem use tensor methods or Semi-definite programming and require assumptions; further the running time is a high polynomial (see [4, 14]). But we can pose the problem of recovery of the $k$ underlying communities differently. Instead of aiming to get ${\bf B}$, we wish to pin down a vector for each community. First we pick at random a subset $V_{1}$ of $d$ people and lets call the remaining set of people $V_{2}$. For convenience, assume $|V_{2}|=n$. We will represent community $\ell$ by a $d$ vector, where the $d$ “features” are the probabilities that a person in $V_{2}$ in community $\ell$ knows each of the $d$ people in $V_{1}$. With a sufficiently large $d$, it is intuitively clear that the community vectors describe the communities well. Letting the columns of a $k\times d$ matrix ${\bf W}^{(1)}$ and the columns of a $k\times n$ matrix ${\bf W}^{(2)}$ denote the fractional membership weights of members in $V_{1}$ and $V_{2}$ respectively, the probability matrix for the bipartite graph on $(V_{1},V_{2})$ is given by

[TABLE]

This reduces the Model Estimation problem here to our geometric problem: Given ${\bf A}$, the adjacency matrix of the bipartite graph, estimate ${\bf M}$. Note that the random variables in ${\bf W}^{(1)},{\bf W}^{(2)}$ are independent.

[1] assumes that each column of ${\bf W}^{(2)}$ is picked from the Dirchlet distribution. An usual setting of the concentration parameter of this Dirichlet is $1/k$ and we will use this value. The proof that Proximate Data Assumption is satisfied is exactly on the same lines as for Topic Models above. The proof that spectral norm of the perturbation matrix ${\bf A}-{\bf P}$ is small again draws on Random Matrix Theory ([23]), but, is slightly simpler. MMSB also shares the four properties discussed in the context of LDA.

Data Outside $K$ We illustatrte in a simple case. Suppose $k=1$ and ${\bf M}$ has all entries equal to $p$. The graph of who knows whom is generally sparse. This means $p\in o(1)$. $A_{\cdot,j}$ now will consist of $pd$ 1’s (in expectation), so we will have $|A_{\cdot,j}-M_{\cdot,1}|\approx\sqrt{pd}$, whereas $|M_{\cdot,\ell}|=p\sqrt{d}$. Since $p\in o(1)$ here, in fact, $|M_{\cdot,1}|$ is $o(1)$ times perturbation, so indeed data is far away from $K$. This example can be generalized to higher $k$. More generally, if block sizes are each $\Omega(d)$ and we have graph sparsity, the same phenomenon happens.

Subset Averages Under the stochastic model, we can again use Random Matrix Theory ([23]) to derive an upper bound on the spectral norm of ${\bf A}-{\bf P}$ similar to LDA. [The proof is different because edges are now mutually independent.] Then as before, we can use Lemma  3.1 to show that for all $R$, the averages of data points and latent points in $R$ are close.

Proximity Since the Drichlet density is used as a prior, we have the same argument as in LDA and one can prove a result similar (2.3).

Input Sparsity The graph of who knows whom is typically sparse.

2.3 Adversarial Clustering

Traditional Clustering problems arising from mixture models can be stated as: Given $n$ data points $A_{\cdot,1},A_{\cdot,2},\ldots,A_{\cdot,n}\in{\bf R}^{d}$ which can be partitioned into $k$ distinct clusters $C_{1},C_{2},\ldots,C_{k}$, find the means $M_{\cdot,1},M_{\cdot,2},\ldots,M_{\cdot,k}$ of $C_{1},C_{2},\ldots,C_{k}$. While there are many results for mixture models showing that under stochastic assumptions, the $M_{\cdot,\ell}$ can be estimated, more relevant to our discussion are the results of [17] and [7], which show that under a deterministic assumption, the clustering problem can be solved. In more detail, letting $P_{\cdot,j}$ denote the mean of the cluster $A_{\cdot,j}$ belongs to (so, $P_{\cdot,j}\in\{M_{\cdot,1},M_{\cdot,2},\ldots,M_{\cdot,k}\}$) and defining $\sigma,\delta$ as follows ($\sigma$ denotes the maximum over directions of the square root of the mean-squared perturbation in the direction, and $\delta$ is a lower bound on the weight of a cluster):

[TABLE]

[17] and [7] show that:

If $|M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}|\geq ck\frac{\sigma}{\sqrt{\delta}}\forall\ell\not=\ell^{\prime},$ the $M_{\cdot,\ell}$ can be found within error $O(\sqrt{k}\sigma/\sqrt{\delta})$.

Note that $\delta$ may go to zero as $n\rightarrow\infty$. We observe (see Lemma  9.1 for a formal statement) that if the error can be improved to $o(\sigma/\sqrt{\delta})$ just in the case when $k=2$, then, we can find $o(\sqrt{n})$ size planted cliques in a random graph $G(n,1/2)$ settling a major open problem. So, at the present state of knowledge, an error of $O(\sigma/\sqrt{\delta})$ (times factors of $k$) is the best dependence of the error on $\sigma,\delta$ we can aim for and our algorithm will achieve this for the LLS problem.

2.3.1 Adversarial Noise

We now allow an adversary to choose for each $\ell\in[k]$, a subset $S_{\ell}$ of $C_{\ell}$ of cardinality $\delta n$ and to add noise $\Delta_{j}$ to each data point $A_{\cdot,j}$, where, the $\Delta_{j}$ satisfy the following conditions:

• •

For all $j$, $P_{\cdot,j}+\Delta_{j}\in\text{Convex Hull of }(M_{\cdot,1},M_{\cdot,2},\ldots,M_{\cdot,k})$ and

• •

For $j\in\cup_{\ell=1}^{k}S_{\ell}$, $|\Delta_{j}|\leq 4\sigma/\sqrt{\delta}$.

In words, each data point $A_{\cdot,j}$ is moved an arbitrary amount towards the convex hull of the means of the clusters it does not belong to (which intuitively makes the learning problem more difficult), but, for $\delta n$ points in each cluster, the move is by distance at most $O(\sigma/\sqrt{\delta})$ . Note that the $C_{\ell},S_{\ell},P_{\cdot,j},\Delta_{j}$, the original $A_{\cdot,j}$ are all latent; only the adversarial-noise-added $A_{\cdot,j}$ are observed and the problem is to find the $M_{\cdot,\ell}$ approximately. For convenience, we pretend that the same noise $\Delta_{j}$ has been added to the latent points $P_{\cdot,j}$ as well, so ${\bf A}-{\bf P}$ remains invariant. Also for ease of notation, we denote the noise-added data points, by $A_{\cdot,j}$ and the noise-added latent points by $P_{\cdot,j}$ from now on.

Now, it is easy to see that (the new) $P_{\cdot,j}$ satisfy the following:

[TABLE]

Also, it is clear that any set of $P_{\cdot,j}$ satisfying these two conditions qualify as a set of latent points for our Adversarial clustering problem.

Notation: $n,d,k$ are reserved for number of data points, number of dimensions of the space and the number of vertics of $K$ respectively. Also, we reserve $i,i^{\prime},i_{1},i_{2}$ to index elements of $[d]$, $j,j^{\prime},j_{1},j_{2}$ to index $[n]$ and $\ell,\ell^{\prime},\ell_{1},\ell_{2}$ to index $[k]$. ${\bf A},{\bf M},{\bf P}$ are reserved for the roles described above. $A_{\cdot,j}$ denotes the $j$ th column of matrix ${\bf A}$ and so too for other matrices. For a vector valued random variable $X\in{\bf R}^{d}$, $\mbox{Var}(X)$ denotes the covariance matrix of $X$. For a matrix ${\bf B}$, $s_{1}({\bf B}),s_{2}({\bf B}),\ldots$ are the singular values arranged in non-increasing order. $||{{\bf B}}||=\mbox{Max}_{x:|x|=1}|x^{T}{\bf B}|=s_{1}({\bf B})$ is the spectral norm. CH denotes convex hull of what follows. CH of a matrix is the convex hull of its columns.

2.4 Latent k-simplex is an unifying model

From the review of existing models one can conclude that indeed learning many latent variable models can be posed as the Latent k-simplex problem. More precisely, the learning problem can be understood as aiming to recover the $k$ vertices $M_{\cdot,1},M_{\cdot,2},\ldots,M_{\cdot,k}$ of a latent simplex $K$ in ${\bf R}^{d}$ given data generated from $K$ under a stochastic or a deterministic model.

In situations with hypothesized stochastic processes, the following assumptions are made on the data generation process (of generating $A_{\cdot,j}$ given $P_{\cdot,j}$):

[TABLE]

Under these conditions, Random Matrix Theory [23] can be used to prove that $||{\bf A}-{\bf P}||$ is bounded: (See for example proof of Lemma 7.1 for a precise statement)

[TABLE]

Also in both LDA and MMSB models, a Dirichlet prior is assumed on the $P_{\cdot,j}$. More precisely, since $P_{\cdot,j}$ are convex combinations of the vertices of $K$, there is a (latent) $k\times n$ matrix ${\bf W}$ with non-negative entries and column sums equal to 1 such that

[TABLE]

The Dirichlet density (under the usual setting of parameters) is maximum at the corners of the simplex and attaches at least a positive fraction of weight to the region close to each corner. Thus, one has with high probability:

[TABLE]

for suitable $\varepsilon_{1},\delta_{1}$.

Here, we do not assume a stochastic model of data generation, neither do we posit any prior on $P_{\cdot,j}$. Instead, we make deterministic assumptions. To impose a bound on $\sigma$, analogous to (2.8), we impose an assumption we call Spectrally bounded Perturbations. Furthermore, to characterize the property that there is concentration of observed data near the extreme points, we make the assumption we call **Proximate  Latent Points **. We will show later that in the LDA model as well as usual MMSB model, our deterministic assumptions are satisfied with high probability.

The deterministic assumptions have another advantage: In Clustering and other situations like NMF, there is geenrally no stochastic model, but we can still apply our results. An upper bound on spectral norm similar to what we use here was used for the special cases of (non-adversarial) clustering and pure mixture models (which are a special case of the ad-mixture models we have here) in [17]. Also in the clustering context, the Proximity assumption is similar to the assumption often made of a lower bound on cluster weights.

The main contribution of this paper is to show that Learning a Latent Simplex (LLS) problem can be solved using three deterministic assumptions: the two described above plus an assumption of Well Separatedness, a standard assumption in mixture models, that the vertices of $K$ are well-separated.

3 Subset Smoothing

We now describe the starting point of our solution method, which is a technique we call “Subset Smoothing”. It shows that the simplex $K$ which we are trying to learn is well-approximated by a data-determined polytope $K^{\prime}$ which is the convex hull of the ${n\choose\delta n}$ points, each of which is the average of $A_{\cdot,j}$ over $j$ in a subset of size $\delta n$. While the description of $K^{\prime}$ is exponential-sized, it is easy to see that $K^{\prime}$ admits a polynomial-time (indeed linear time) optimization oracle and this will be our starting point. First, to see why averages of $\delta n$ subsets of data help, note that our assumptions will not place any upper on individual perturbations $|A_{\cdot,j}-P_{\cdot,j}|$; indeed they are typically very large in Latent Variable Models as we discussed above. However, we will place an upper bound on $||{\bf A}-{\bf P}||$, the spectral norm of the perturbation matrix or equivalently on $\sigma=||{\bf A}-{\bf P}||/\sqrt{n}$. [Such an upper bound on $||{\bf A}-{\bf P}||$ is usually available in stochastic latent variable models via Random Matrix Theory, as we will see later in the paper for LDA and MMBM.] For any subset $R$ of $[n]$, denote by $A_{\cdot,R}$ the average of $A_{\cdot,j},j\in R$, and so for ${\bf P}$, namely

[TABLE]

It is easy to see that an upper bound on $\sigma$ (defined in (2.5)) implies an upper bound on $\mbox{Max}_{R:|R|=\delta n}|A_{\cdot,R}-P_{\cdot,R}|$:

Lemma 3.1

For all $S\subseteq[n],$ $|A_{\cdot,S}-P_{\cdot,S}|\leq\frac{\sigma\sqrt{n}}{\sqrt{|S|}}.$

• Proof.

This just follows from the fact that $|A_{\cdot,S}-P_{\cdot,S}|=\frac{1}{|S|}|({\bf A}-{\bf P}){\bf 1}_{S}|\leq\frac{1}{|S|}||{\bf A}-{\bf P}||\;|{\bf 1}_{S}|$ and $|{\bf 1}_{S}|=\sqrt{|S|}$.

Now, we can prove that the data-determined polytope $K^{\prime}=CH(A_{\cdot,S}:|S|=\delta n)$ is close to the simplex $K=CH({\bf M})$ which we seek to find. Closeness of two sets $K_{1},K_{2}$ is measured in Hausdorff metric $D(K_{1},K_{2})$ which we define here. For sets $K_{1},K_{2}$, define:

[TABLE]

Note Dist$(K_{1},K_{2})$ may not equal Dist$(K_{2},K_{1})$. If $K_{1}$ is a single point $x$, we write $\text{Dist}(x,K_{2})$. $\text{Dist}(x,K_{2})$ is a convex function of $x$. So, we have:

Claim 3.1

If $x$ varies over polytope $K_{1}$, then the maximum of $\text{Dist}(x,K_{2})$ is attained at a vertex of $K_{1}$.

Hausdorff diatsnce $D$ is defined by $D(K_{1},K_{2})=\mbox{Max}(\text{Dist}(K_{1},K_{2}),\text{Dist}(K_{2},K_{1}))$.

Theorem 3.1

Let $K^{\prime}=CH(A_{\cdot,R}:|R|=\delta n)$. We have $D(K,K^{\prime})\leq 5\sigma/\sqrt{\delta}.$

• Proof.

We first prove that Dist$(K,K^{\prime})\leq 5\sigma/\sqrt{\delta}$ for which, by Claim (3.1), it suffices to show that $\forall\ell\in[k],\exists S\subseteq[n],|S|=\delta n:|M_{\cdot,\ell}-A_{\cdot,S}|\leq\frac{5\sigma}{\sqrt{\delta}}$. Take $S=S_{\ell}$ (for the $S_{\ell}$ defined in (2.7)). Since for each $j\in S_{\ell}$, we have $|P_{\cdot,j}-M_{\cdot,\ell}|\leq 4\sigma/\sqrt{\delta}$, we have by convexity of $|\cdot|$ that $|P_{\cdot,S_{\ell}}-M_{\cdot,\ell}|\leq 4\sigma/\sqrt{\delta}$. By Lemma (3.1), it follows that $|P_{\cdot,S_{\ell}}-A_{\cdot,S_{\ell}}|\leq\sigma/\sqrt{\delta}$. Adding these two and using the triangle inequality, we get $|M_{\cdot,\ell}-A_{\cdot,S_{\ell}}|\leq\frac{5\sigma}{\sqrt{\delta}}$ as claimed.

To prove that Dist$(K^{\prime},K)\leq 5\sigma/\sqrt{\delta}$, again by Claim (3.1), it suffices to show for any $S\subseteq[n]$, $|S|=\delta n$, Dist$(A_{\cdot,S},K)\leq\sigma\sqrt{\delta}$. Note that $P_{\cdot,S}\in K$ (since it is the avergae of $\delta n$ points in $K$) and also $|A_{\cdot,S}-P_{\cdot,S}|\leq\sigma/\sqrt{\delta}$ by Lemma (3.1) and so Dist$(A_{\cdot,S},K)\leq\sigma/\sqrt{\delta}$.

Lemma 3.2

Given any $u\in{\bf R}^{d}$, $\mbox{Max}_{x\in K^{\prime}}(u\cdot x)$ can be found in linear time (in ${\bf A}$).

• Proof.

One computes $u\cdot A_{\cdot,j},j=1,2,\ldots,n$ by doing a matrix-vector product in time $O(\text{nnz}({\bf A}))$ and takes the avrage of the $\delta n$ highest values.

The above immediately suggests the question: Can we just optimize $k$ linear functions over $K^{\prime}$ and hope that each optimal solution gives us an approximation to a new vertex of $K$? It is easy to see that if we choose the $k$ linear functions at random, the answer is not necessarily. However, this idea does work with a careful choice of linear functions. We will now state the choices which lead to our algorithm.

4 Statement of Algorithm

Our algorithm will choose (carefully) $k$ linear functions. We will show that optimizing each of these will give us an approximation to a new vertex of $K$, thus at the end, we will have all $k$ vertices. The algorithm can be stated in a simple self-contained way and we do so presently. We will prove correctness under our assumptions after formalizing the assumptions. However, the proof is not nearly as simple as the algorithm statement and will occupy the rest of the paper. Of the steps in the algorithm, the truncated SVD step at the start is costly and does not meet our time bound. We will later replace it by the classical subspace power iteration method which does.

5 Learning a Latent k-simplex (LLS) problem and Main results-Informal statements

Before we state our results we informally describe the main results of the paper.

Recall that the Latent k-simplex problem: Given data points $A_{\cdot,j},j=1,2,\ldots,n\in{\bf R}^{d}$, obtained by perturbing latent points $P_{\cdot,j},j=1,2,\ldots,n$ respectively, from a latent $k-$simplex $K$, learn the $k$ vertices. Our main result is that there is a quasi-input sparsity time algorithm which could solve this problem under certain assumptions.

Assumptions:

We will informally introduce the assumptions to explain our results.

• •

Well-Separatedness(Informal Statement) Each of the $M_{\cdot,l}$ for $\ell=1,2,\ldots,k$, has a substantial component orthogonal to the space spanned by the other $M_{\cdot,\ell^{\prime}}$. This makes the vectors well separated.

• •

**Proximate  Latent Points **(Informal Statement) $\forall\ell\in[k]$, there are at least $\delta n$ $j$ ’s with $P_{\cdot,j}$ close to (at distance at most $4\sigma/\sqrt{\delta}$) from $M_{\cdot,\ell}$. $\delta$ is in $(0,1)$ and can depend on $n,d$, in particular going to zero as $n,d\rightarrow\infty$. Note that in the case of $k-$means Clustering, all data have $P_{\cdot,j}=$ a vertex of $K$; there, $\delta$ is the minimum fraction of data points in any cluster.

• •

Spectrally bounded perturbations(Informal Statement) We assume

[TABLE]

It is clear that if the perturbations are unbounded then it is impossible to recover the true polytope. We have already discussed above why the upper bound on $\sigma$ is reasonable.

Now we can state the main problem and result.

Learning a Latent Simplex (LLS) problem

(Informal Statement) * Given $n$ data points $A_{\cdot,j},j=1,2,\ldots,n\in{\bf R}^{d}$ such that there is an unknown $k-$simplex $K$ and unknown points $P_{\cdot,j}\in K,j=1,2,\ldots,n$, find approximations to vertices of $K$ within error poly$(k)\sigma/\sqrt{\delta}$. [I.e., find $\widetilde{M}_{\cdot,\ell},\ell=1,2,\ldots,k$ such that there is some permutation of indices with $|M_{\cdot,\ell}-\widetilde{M}_{\cdot,\ell}|\leq\text{poly}(k)\sigma/\sqrt{\delta}\;\text{for}\;\ell=1,2,\ldots,k.$ ]*

The main result can be informally stated as follows:

Theorem 5.1

*(Informal Statement) If observations, generated through Spectrally bounded Perturbation of latent points generated from a polytope with well-separated vertices, satisfy the ***Proximate  Latent Points **assumption, then the main problem can be solved in time $O^{*}(k\times\mbox{nnz}({\bf A})+k^{2}d)$. 111$O^{*}$ hides logarithmic factors in $n,k,d$ as well as factors in $\delta,\alpha$.

We will develop an algorithm which approximately recovers the vertices of the Latent k-simplex and achieves the run-time complexity mentioned in the theorem.

Literature related to learning simplices

In Theoretical Computer Science and Machine Learning, there is substantial literature on Learning Convex Sets [20, 16], intersection of half spaces [15, 20, 21], Parallelopipeds[13] and simplices [5] However, this literature does not address our problem since it assumes we are given data points which are all in the convex set, whereas, in our settings, as we saw, they are often (far) outside.

There are a number of algorithms in Unsupervised Learning as mentioned above. But algorithms with proven time bounds have two issues which prevent their use in our problem: (a) all of them depend on context-specific technical assumptions and (b) They have worse time bounds. The quasi-input-sparsity complexity is a very attractive feature of our algorithm and the generality of the problem makes it applicable to wide range of latent variable models such as MMSB, Topic Models, and Adversarial Clustering. It is to be noted that our method also gives an immediate quasi-input-sparsity algorithm for $k-$ means clustering for $k\in O^{*}(1)$. We are not aware of any such result in the literature (see Section  6.2.1).

6 Assumptions, Subset Smoothing, and Main results

In this section we formally describe the key assumptions, and our main results. We derive an algorithm which uses subset smoothing and show that it runs in quasi-input sparsity time.

6.1 Assumptions

As informally introduced before, the three main assumptions are necessary for the development of the algorithm. They crucially depend on the following parameters.

• •

The Well-separatedness of the model depends on $\alpha$, which is a real number in $(0,1)$. We assume under Well-separatredness that each $M_{\cdot,\ell}$ has a substantial component, namely, $\alpha\mbox{Max}_{\ell^{\prime}}|M_{\cdot,\ell^{\prime}}|$, orthogonal to the span of the other $M_{\cdot,\ell^{\prime}}$. Note that of course, the component of $M_{\cdot,\ell}$ orthogonal to span of other $M_{\cdot,\ell^{\prime}}$ cannot be greater than $|M_{\cdot,\ell}$, so implicit in this assumption, we require all $|M_{\cdot,\ell^{\prime}}$ to be within $\alpha$ factor of each other. $\alpha$ is an arbitrary model-determined parameter, so it can depend on $k$, but not on $n,d$. Higher the value of $\alpha$ the more well-separated the model is.

• •

The parameter $\delta\in(0,\frac{1}{k})$, quantifies the fraction of data close to each of the vertices. **Proximate  Latent Points **assumption requires that for each $\ell$, $\delta$ fraction of the $n$ latent points lie close to each vertex of $K$.

Assumption 1

Well-Separatedness* We assume that there is an $\alpha\in(0,1)$ such that ${\bf M}$ matrix obeys the following*

[TABLE]

Well-Separatedness is an assumption purely on the model ${\bf M}$ and is not determined by the data.

Assumption 2

**Proximate  Latent Points **: The model satisfies ${\bf Proximate~{}LatentPoints~{}}$ assumption if

[TABLE]

• Remark:

Note that $\delta$ is always at most $1/k$ and is allowed to be smaller, it is allowed to go to zero as $n\rightarrow\infty$

Assumption 3

Spectrally Bounded Perturbations* The following relationship will be assumed,*

[TABLE]

This assumption depends on the observed data and somewhat weakly on the model. The reader may note that in pure mixture models, like Gaussian Mixture Models, a standard assumption is: Component means are separated by $\Omega^{*}(1)$ standard deviations. (6.12) is somewhat similar, but not the same: while we have $\mbox{Min}_{\ell}|M_{\cdot,\ell}|$ on the right hand side, the usual separation assumptions would have $\mbox{Min}_{\ell\not=\ell^{\prime}}|M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}|$. While these are similar, they are incomparable.

• Remark:

It is important that we only have poly$(k)$ factors and no factor dependent on $n,d$ in the denominator of the right hand side. Since $n,d$ are larger than $k$, a dependence on $n,d$ would have been too strong a requirement and generally not met in applications. Of course our dependence on $k$ could use improvement. The factor of $\sigma/\sqrt{\delta}$ seems to be necessary at the current state of knowledge, otherwise one can solve the planted clique problem in $o(\sqrt{n})$ regime in polynotmal time. A formal statement is provided in Lemma  9.1

6.2 A Quasi Input Sparsity time Algorithm for finding extreme points of K

In this subsection we present an algorithn based on the Assumptions and Subset smoothing described earlier. The algorithm is the same as described in Section  3, but with the first step of computing the exact truncated SVD replaced by the classical subspace power iteration which meets the time bounds. The algorithm proceeds in $k$ stages (recall $k$ is the number of vertices of $K$), in each stage maximizing $|u\cdot x|$ over $x\in K^{\prime}$ for a carefully chosen $u$. The maximization can be solved by just finding all the $u\cdot A_{\cdot,j}$ and taking the largest (or smallest) $\delta n$ of them. Unlike the algorithm, the proof of correctness is not so simple. Among the tools it uses are the $sin-\Theta$ theorem in Numerical Analysis, an extension which we prove, and the properties of random projections. A brief introduction to this and some basic properties may be found again in Section 8.

Theorem 6.1

*Suppose we are given $k\geq 2$ and data ${\bf A}$, satisfying the assumptions of Well-Separatedness (6.10), ***Proximate  Latent Points **(6.11), and Spectrally Bounded Perturbations (6.12). Then, in time $O^{*}\left(k\left(\text{nnz}({\bf A})+kd\right)\right)$ time, the Algorithm LKS finds subsets $R_{1},R_{2},\ldots,R_{k},$ of cardinality $\delta n$ each such that after a permutation of columns of ${\bf M}$, we have with probability at least $1-(c/\sqrt{k})$:

[TABLE]

We next state the main result which directly implies theorem (6.1). The hypothesis of the result below is that we have already found $r\leq k-1$ columns of ${\bf M}$ approximately, in the sense that we have found $r$ subsets $R_{1},R_{2},\ldots,R_{r}\subseteq[n],|R_{t}|=\delta n$ so that there are $r$ distinct columns $\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$ of ${\bf M}$ with $M_{\cdot,\ell_{t}}\approx A_{\cdot,R_{t}}$ for $t=1,2,\ldots,r$. The theorem gives a method for finding a $R_{r+1},|R_{r+1}|=\delta n$ with $A_{\cdot,R_{r+1}}\approx M_{\ell}$ for some $\ell\notin\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$.

Theorem (6.1) follows by applying Theorem (6.2) $k$ times.

Theorem 6.2

*Suppose we are given data ${\bf A}$ and $k\geq 2$ satisfying the assumptions of Well-Separatedness (6.10), ***Proximate  Latent Points **(6.11) and Spectrally Bounded Perturbations (6.12). Let $r\leq k-1$. Suppose $R_{1},R_{2},\ldots,R_{r}\subseteq[n]$, each of cardinality $\delta n$ have been found and are such that there exist $r$ distinct elements $\ell_{1},\ell_{2},\ldots,\ell_{r}\in[k]$, with:222We do not know ${\bf M}$ or $\ell_{1},\ell_{2},\ldots,\ell_{r}$, only their existence is known.

[TABLE]

Let $V$ be any $k-$ dimensional subspace of ${\bf R}^{d}$ with $\sin\Theta(V,\mbox{Span}(v_{1},v_{2},\ldots,v_{k}))\leq\sigma/\sqrt{\delta}$ (where, $v_{1},v_{2},\ldots,v_{k}$ are the top $k$ left singular values of ${\bf A}$). Suppose $u$ is a random unit length vector in the $k-r$ dimensional sub-space $U$ given by:

[TABLE]

and suppose

[TABLE]

Then, with probability at least $1-(c/k^{3/2})$,

[TABLE]

Further this can be carried out in time $O^{*}(\text{nnz}({\bf A})+dk)$ time.

6.2.1 Quasi-Input-Sparsity Based Complexity

While there has been much progress on devising algorithms for sparse matrices and indeed for Low-Rank Approximation (LRA) nearly optimal dependence of $O^{*}(\text{nnz})$ on input sparsity is known [11], there has not been such progress on standard $k-$means Clustering (for which several algorithms first do LRA). This is in spite of the fact that there are many instances where the data for Clustering problems is very sparse. For example, Graph Clustering is a well-studied area and many graphs tend to be sparse. Our complexity dependence on nnz is $k\times{\tt nnz}$, and hence we refer it as quasi-input-sparsity time complexity when $k\in O^{*}(1)$. We do not have a proof of a corresponding lower bound. But recently,in [19], it was argued that for kernal LRA, $k$ nnz is possibly optimal unless matrix multiplication can be improved. We leave as an open problem the optimality of our nnz dependence. We are unaware of an algorithm for any of the special cases, considered in this paper, has a better complexity than ours.

7 Latent Variable Models as special cases

In this section we discuss three latent variable models LDA, MMSB and Adversarial Clustering and prove that they are special cases of our general geometric problem.

7.1 LDA as a special case of Latent k-simplex problem:

In the LDA setup we will consider that the prior is $\mbox{Dir}(\frac{1}{k},k)$ on the unit simplex. The following arguments apply to Dir$(\beta,k)$ for any $\beta\leq 1/k$, but we do the case when $\beta=1/k$ here. We also assume, what we call “lumpyness” of $M_{\cdot,\ell}$ which intuitively says that for any $\ell$, $M_{i,\ell},i=1,2,\ldots,d$ should not all be small, or in other words, the vector $M_{\cdot,\ell}$ should be “lumpy”. We assume that

[TABLE]

This assumption is consistent with existing literature. It is common practice in Topic Modelling to assume that in every topic there are a few words which have very high probability. A topic with weights distributed among all (or many) words is not informative about the theme. Furthermore, if word frequencies satisfy power law, it is indeed easy to see this assumption holds. It is to be noted, a weaker assumption than power law, namely, that the $O(1)$ highest frequency words together have $\Omega(1)$ frequency, is also enough to imply our assumption that $|M_{\cdot,\ell}|\in\Omega(1)$, as is easy to see by summing. If even this last assumption is violated, it means say that a large number of high frequency words still do not describe the topic well which wouldn’t make for a reasonably interpretable topic model.

Lemma 7.1

Suppose ${\bf A},{\bf P}$ are as above. Assume that $|M_{\cdot,\ell}|\in\Omega(1)$ for all $\ell$. Suppose $W_{\cdot,j},j=1,2,\ldots,n$ are i.i.d. distributed according to Dir$(k,1/k)$ and assume $m,n$ are at least a sufficiently large polynomial function of $\frac{k}{\alpha}$. Let $\delta=\frac{c\sigma}{\sqrt{k}}$. Then, (6.11) and (6.12) are satisfied with high probability.

• Proof.

See Section 9.1.

• Remark:

The only assumption for which we have made no assertion is Well-Separatedness (6.10). This is because, there is no generally assumed prior for generating ${\bf M}$. If one were to assume a Dirichlet prior for ${\bf M}$ with sufficiently low concentration parameter or assume a power law frequency distribution of words, one can show (6.10) holds.

Theorem 7.1

Suppose ${\bf A},{\bf P}$ are as in Lemma  7.1. Assume that $|M_{\cdot,\ell}|\in\Omega(1)$ for all $\ell$ and $n\geq m$. Suppose $W_{\cdot,j},j=1,2,\ldots,n$ are i.i.d. distributed according to Dir$(k,1/k)$ and assume $m,n$ are at least a sufficiently large polynomial function of $k/(\alpha)$. Let $\delta=\frac{c\sigma}{\sqrt{k}}$. Also assume the Well-Separatedness assumption (6.10) is satisfied. Then, our algorithm with high probability finds approximations $\widetilde{M}_{\cdot,1},\widetilde{M}_{\cdot,2},\ldots,\widetilde{M}_{\cdot,k}$ to the topic vectors so that (after a permutation of indices),

[TABLE]

7.2 MMSB Models

The assertions and proofs here are similar to LDA. The difference is in the proof of an upper bound on $\sigma$ (Spectrally Bounded Perturbations), since, here, all the edges of the graph (entries of ${\bf A}$) are mutually independent, but there is no effective absolute (probability 1) bound on perturbations.

${\bf A},{\bf M},{\bf P},{\bf W^{(2)}}$ have the meanings discussed in Section  2.2, see equation (2.4). We introduce one more symbol here: we let $\nu$ denote the maximum expected degree of any node in the bipartite graph, namely,

[TABLE]

Instead of the “lumpyness” assumption that $|M_{\cdot,\ell}|\in\Omega(1)$ we made in Topic Modeling,, we make the assumption here that $|M_{\cdot,\ell}|\geq\nu^{1/8}$. The reader can verify that this won’t be satisfied if $\nu$ is small and $P_{ij}$ are spread out, but will be if $\nu$ is at least $d^{\gamma}$ for a small $\gamma$.

Lemma 7.2

Suppose ${\bf A},{\bf P}$ are as above. Assume $|M_{\cdot,\ell}|\geq\nu^{1/8}$ for all $\ell$ and $n\geq d$. Also suppose $n/d$ is a sufficiently high polynomial in $\frac{k}{\alpha}$.. Suppose $W_{\cdot,j}^{(2)}$ are i.i.d. distributed according to $Dir(k,1/k)$. Let $\delta=c\sigma/\sqrt{k}$. Then, (6.11) and (6.12) are satisfied with high probability.

Theorem 7.2

Suppose ${\bf A},{\bf P}$ are as in Lemma  7.2. Assume $|M_{\cdot,\ell}|\geq\nu^{1/8}$ for all $\ell$ and $n\geq d$. Also suppose $n/d$ is a sufficiently high polynomial in $k/\alpha$, where, $\varepsilon\in[0,1]$. Suppose $W_{\cdot,j}^{(2)}$ are i.i.d. distributed according to $Dir(k,1/k)$. Let $\delta=c\sigma/\sqrt{k}$. Suppose in addition, (6.10) holds. Then the algorithm finds $\widetilde{M}_{\cdot,\ell},\ell=1,2,\ldots,k$ such that after a permutation, we have (whp)

[TABLE]

• Remark:

By making $n$ sufficiently larger than $\nu,d$, we can make the error small.

7.3 Adversarial Clustering

There is a latent ground-truth $k-$Clustering $C_{1},C_{2},\ldots,C_{k}$ with cluster centers $M_{\cdot,1},M_{\cdot,2},\ldots,M_{\cdot,k}$. There are $n$ latent points $P_{\cdot,j},j=1,2,\ldots,n$ with $P_{\cdot,j}=M_{\cdot,\ell}$ for all $j\in C_{\ell}$. The following assumptrions are satisfied:

Well-Separatedness condition (6.10). 2. 2.

Each cluster has at least $\delta n$ data points. 3. 3.

Spectrally bounded perturbations (6.12).

We then allow an adversary to introduce noise as in Section  2.3.1.

Theorem 7.3

Given as input data points after adversarial noise as above has been introduced, our algorithm finds (the original) $M_{\cdot,\ell}$ to within error as in Theorem (6.1).

• Remark:

We want to point out that the traditional $k-$means clustering does not solve the Adversarial Clustering problem. A simple example in one dimension is: The original $K$ is $[-1,+1]$ and $n/2$ points are in each cluster with a small $\sigma$. We then move $n(0.5-\delta)$ from the cluster centered at -1 each by +0.5 and $n(0.5-\delta)$ points from cluster centered at +1 by -.5 each. It is easy to see that the best $2-$means clustering of the noisy data is to locate two cluster centers, one near each of $\pm.5$, (depending on $\delta$), not near $\pm 1$.

8 Closeness of Subspaces and Subspace power iteration

In this section we will present the classical Subspace power iteration algorithm which finds an approximation to the subspace spanned by the top $k$ (left) singular vectors of ${\bf A}$. It has a well-known elegant proof of convergence, which also we present here, since, usual references often present more general (and also more complicated) proofs. Let

[TABLE]

8.1 Closeness of Subspaces

First, we recall a measure of closeness of sub-spaces. Numerical Analysis has developed, namely, the notion of angles between sub-spaces, called Principal angles. Here, we need only one of the principal angles which we define now.

For any two sub-spaces $F,F^{\prime}$ of ${\bf R}^{d}$, define

[TABLE]

[TABLE]

The following are known facts about $\sin\Theta$ function: If $F,F^{\prime}$ have the same dimension and the columns of ${\bf F}$ (respectively ${\bf F}^{\prime}$) form an orthonormal basis of $F$ (respectivel $F^{\prime}$), then

[TABLE]

where, the columns of matrix ${\bf G}$ form a basis for $F^{\perp}$, and assuming the inverse of ${\bf F}^{T}{\bf F}^{\prime}$ exists.

An important Theorem due to Wedin[24], also known as the $\sin\Theta$ theorem, proves a bound on the $\sin\Theta$ between SVD-subspaces of a matrix and its perturbation:

Theorem 8.1

[24]** Suppose ${\bf R},{\bf S}$ are any two $d\times n$ matrices. Let $m\leq\ell$ be any two positive integers. Let $S_{m}({\bf R}),S_{\ell}({\bf S})$ denote respectively the subspaces spanned by the top $m$, respectively top $\ell$ left singular vectors of ${\bf R}$, respectively, ${\bf S}$. Suppose $\gamma=s_{m}({\bf R})-s_{\ell+1}({\bf S})>0$. Then,

[TABLE]

Corollary below says the SVD subspace of ${\bf A}$ is approximately equal to the Span of ${\bf P}$.

Corollary 8.1

Let ${\bf A},{\bf P}$ be defined in Section  2.4.

[TABLE]

• Proof.

Apply Theorem (8.1) with ${\bf R}={\bf A},{\bf S}={\bf P},m=\ell=k$

We next find SVD subspace of ${\bf A}$ by subspace power iteration.

8.2 Subspace-Power Iteration

Recall that $v_{1},\ldots,v_{d}$ are left singular vectors of ${\bf A}$ and ${\bf Q}_{T}$ is the $T$ iterate of the subspace power iteration.

Theorem 8.2

Convergence of Subspace power iteration**

[TABLE]

• Proof.

Let ${\bf F},{\bf G}$ be the $d\times k,d\times(d-k)$ matrices with columns $v_{1},v_{2},\ldots,v_{k}$, respectively $v_{k+1},v_{k+2},\ldots,v_{d}$ and $F,G$ be respectively the subspaces spanned by their columns. Note that from (8.1), we have

[TABLE]

Lemma 8.1

[TABLE]

• Proof.

${\bf Q_{t}}={\bf Z_{t}}{\bf R}$, where ${\bf R}$ is the invertiblke matrix of Grahm-Scmidt orthogonalization, We write

Define $DS_{i:j}=\mbox{Diag}(s_{i}^{2}({\bf A}),s_{i+1}^{2}({\bf A}),\ldots s_{j}({\bf A})^{2})$ for positive integers $i,j$ such that $i<j$.

[TABLE]

since, ${\bf G}^{T}{\bf Z_{t}}={\bf G}^{T}{\bf A}{\bf A}^{T}{\bf Q_{t-1}}={\bf G}^{T}\sum_{t=1}^{d}s_{t}^{2}({\bf A})v_{t}v_{t}^{T}{\bf Q_{t-1}}=DS_{k+1:d}{\bf G}^{T}{\bf Q_{t-1}}$ and similarly, ${\bf F}^{T}{\bf Z_{t}}=DS_{1:k}{\bf F}^{T}{\bf Q_{t-1}}$. This proves the Lemma by induction on $t$.

We prove in Claim  8.1 that $s_{k}({\bf P})\geq 4\sigma\sqrt{n}$. Also, $s_{k+1}({\bf A})\leq s_{k+1}({\bf P})+||{\bf A}-{\bf P}||=\sigma\sqrt{n}$. Thus, we have $s_{k+1}/s_{k}\leq 1/2$ and now applying Lemma above, Theoirem (8.2) follows, since for a random choice of ${\bf Q_{0}}$, we have $||{\bf G}^{T}{\bf Q_{0}}\left({\bf F}^{T}{\bf Q_{0}}\right)^{-1}||\leq\mbox{poly}(nd)$.

Next, we apply Wedin’s theorem to prove Lemma (8.2) below which says that any $k$ dimensional space $V$ with small $\sin\Theta$ distance to Span$(v_{1},v_{2},\ldots,v_{k})$ also has small $\sin\Theta$ distance to Span$({\bf M})$. We first need a technical Claim.

Claim 8.1

Recall that $s_{t}$ denotes the $t$ th singular value.

[TABLE]

• Proof.

$s_{k}({\bf M})=\mbox{Min}_{x:|x|=1}|Mx|$. For any $x,|x|=1$, there must be an $\ell$ with $|x_{\ell}|\geq 1/\sqrt{k}$. Now, $|{\bf M}x|\geq|proj({\bf M}x,\mbox{Null}({\bf M}\setminus M_{\cdot,\ell}))|=|x_{\ell}||proj(M_{\cdot,\ell},\mbox{Null}({\bf M}\setminus M_{\cdot,\ell}))|\geq\alpha|M_{\cdot,\ell}|/\sqrt{k}\geq 1000k^{9}\sigma/(\alpha^{2}\sqrt{\delta}\sqrt{k})$ by (6.10) and (6.12) proving the first assertion of the Claim.

Now, we prove the second. Recall there are sets $S_{1},S_{2},\ldots,S_{k}\subseteq[n]$ with $\forall j\in S_{\ell},|P_{\cdot,j}-M_{\cdot,\ell}|\leq\frac{4\sigma}{\sqrt{\delta}}$. We claim the $S_{\ell}$ are disjoint: if not, say, $j\in S_{\ell}\cap S_{\ell^{\prime}}$. Then, $|P_{\cdot,j}-M_{\cdot,\ell}|,|P_{\cdot,j}-M_{\cdot,\ell^{\prime}}|\leq 4\sigma/\sqrt{\delta}$ implies $|M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}|\leq 8\sigma/\sqrt{\delta}\leq\alpha^{2}\mbox{Min}_{\ell}|M_{\cdot,\ell}|/100k^{9}$ by (6.12). But, by (6.10), $|M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}|\geq|\mbox{Proj}(M_{\cdot,\ell},\mbox{Null}({\bf M}\setminus M_{\cdot,\ell})|\geq\alpha|M_{\cdot,\ell}|$ producing a contradiction.

Let ${\bf P}^{\prime}$ be a $d\times k\delta n$ sub-matrix of ${\bf P}$ with its columns $j\in S_{1}\cup S_{2}\cup\ldots\cup S_{k}$ and let ${\bf M}^{\prime}$ be the $d\times k\delta n$ matrix with $M^{\prime}_{\cdot,j}=M_{\cdot,\ell}$ for all $j\in S_{\ell},\ell=1,2,\ldots,k$. We have $s_{k}({\bf M}^{\prime})\geq\sqrt{\delta n}s_{k}({\bf M})\geq 1000k^{8.5}\sqrt{n}\sigma/\alpha^{2}$. Now, $||{\bf P}^{\prime}-{\bf M}^{\prime}||\leq\sqrt{k\delta n}4\sigma/\sqrt{\delta}$. Since $s_{k}({\bf P})\geq s_{k}({\bf P}^{\prime})\geq s_{k}({\bf M}^{\prime})-||{\bf P}^{\prime}-{\bf M}^{\prime}||$, the second part of the claim follows.

Lemma 8.2

Let $v_{1},v_{2},\ldots,v_{k}$ be the top $k$ left singular vectors of ${\bf A}$. Let $V$ be any $k-$ dimensional sub-space of ${\bf R}^{d}$ with

[TABLE]

For every unit length vector $x\in V$, there is a vector $y\in\mbox{Span}({\bf M})$ with

[TABLE]

• Proof.

Since $\mbox{Span}({\bf P})\subseteq\mbox{Span}({\bf M})$, it suffices to prove the Lemma with $y\in$ Span$({\bf P})$. Corollary  8.1 implies:

[TABLE]

[TABLE]

Now, $\sin\Theta(V,\mbox{Span}(v_{1},v_{2},\ldots,v_{k}))\leq\alpha^{2}/1000k^{9}$ and, $\sin\Theta(V,\mbox{Span}({\bf M}))\leq\sin\Theta(V,\mbox{Span}(v_{1},v_{2},\ldots,v_{k}))+\sin\Theta(\mbox{Span}(v_{1},v_{2},\ldots,v_{k}),\mbox{Span}({\bf M}))$, which together imply the Lemma. [We have used here the triangle inequality for $\sin\Theta$ which follows directly from the definition of $\sin\Theta$.]

9 Technical Lemmas

In this section we prove technical claims which are useful to support the main claims of the paper. We begin by noting an useful connection between the choice of $\delta$ and the famous planted clique problem.

Lemma 9.1

Planted Clique and Choice of $\delta$* Suppose $K$ has just two vertices with one of them being the origin and the other equal to ${\underline{1}}_{Q}$ for an unknown subset $Q$ of $[n]$ with $|Q|=q=\delta n$. Let ${\bf P},{\bf A},\sigma$ be as in our notation and suppose a susbet $Q^{\prime}$ of $[n]$ with $|Q^{\prime}|=q$ have each $P_{\cdot,j}$ equal to ${\underline{1}}_{Q}$ and the rest $n-q$ of the $P_{\cdot,j}=$ the origin. Suppose ${\bf A}$ is a random $\pm 1$ matrix with*

[TABLE]

If ${\underline{1}}_{Q}$ can be found within error at most $\varepsilon\sigma/\sqrt{\delta}$, and $|Q|\geq 10\sqrt{\varepsilon}\sqrt{n}$, then, $Q$ can be found exactly.

• Proof.

$E(A_{ij})=P_{ij}$ and $E(A_{ij}-P_{ij})^{2}\leq 2$ as is easy to check. Al;so $A_{ij}$ are mutually independnet. This by Ranodm Matrix Theory ([23]) implies that $\sigma\leq 4$. $\delta=q/n$. Now the conclusion follows from the results of [12].

9.1 Topic Models obey **Proximate  Latent Points **and bounded perturbation assumption

We present the proof of Lemma 7.1 and Theorem 7.1. First, we prove that the Spectrally Bounded Perturbations hold.

• Proof.

(of Lemma  7.1) Note that $|A_{\cdot,j}-P_{\cdot,j}|\leq||A_{\cdot,j}-P_{\cdot,j}||_{1}\leq 2$. Let ${\bf\Sigma_{j}}=E((A_{\cdot,j}-P_{\cdot,j})(A_{\cdot,j}-P_{\cdot,j})^{T})$ be the covariance matrix of $A_{\cdot,j}$ and let ${\bf\Sigma}=\frac{1}{n}{\bf\Sigma_{j}}$. From Theorem 5.44 of [23], we get that with probability at least $1-\varepsilon$,

[TABLE]

where, $c$ includes factors in $1/\varepsilon$. The higher order term here is $\sqrt{||\Sigma||}$ which the following lemma bounds.

Lemma 9.2

With high probability, $\sqrt{||{\bf\Sigma}||}\leq\frac{c}{\sqrt{m}}.$

• Proof.

Let $X_{ijt}=1$ or 0 according as the $t$ th word of document $j$ is the $i$ vocabulary word or not.

[TABLE]

[TABLE]

using distribution of $X_{ijt}$.

[TABLE]

[TABLE]

In the hypothesis of $\delta=c\sigma/\sqrt{k}$ in Theorem (7.1), the following inequality implies (6.12):

[TABLE]

This is in turn implied by the follwng:

[TABLE]

which we now prove by showing that each of the two terms on the lhs is at most 1/2 the rhs. Lemma (9.2) plus the hypothesis that $m$ is a sufficiently large polynomial in $k$ and $|M_{\cdot,\ell}|\in\Omega(1)$ shows the desired upper bound on $\sqrt{||\Sigma||}$. So, it only remains to bound the lower order term, namely, prove that $\frac{c}{\sqrt{n}}\leq\frac{c\alpha^{6}\varepsilon^{4}\mbox{Min}_{\ell}|M_{\cdot,\ell}|^{2}}{10^{6}k^{17}}.$ This follows by noting that $n$ is at least a sufficiently high polynomial in $k$, and $|M_{\cdot,\ell}|\in\Omega(1)$ which proves (6.12).

Now, we turn to proving the (6.11) assumption. For this, we first need the following fact about the Dirichlet density.

Lemma 9.3

If $x$ is a random $k-$ vector picked according to the Dir$(1/k,k)$ density on $\{x:x_{\ell}\geq 0;\sum_{\ell}x_{\ell}=1\}$, then for any $\zeta\in[0,1]$, we have

[TABLE]

• Proof.

The marginal density $q(x_{1})$ of the first coordinate of $x$ is easily seen to be

[TABLE]

where, the normalizing constant $c\geq 1/k$. For $y\in(0,1),y^{(1/k)-1}\geq 1$, so $q(x_{1})\geq\frac{1}{k}(1-x_{1})^{1-(1/k)}$. Now integrating over $x_{1}\in[1-\zeta,1]$, we get the lemma.

Thus, with $\delta=c\sigma/\sqrt{k}$ as assumed in Lemma 7.1, we get that the $S_{\ell}$ defined in (6.11) satisfies $|S_{\ell}|\geq\delta n$, using Höffding-Chernoff bounds. This finishes the proof of Lemma 7.1.

• Proof.

(of Theorem 7.1) Note that its hypothesis also assumes (6.10). So Theorem 6.1 implies that the algorithm finds $\widetilde{M}_{\cdot,\ell},\ell\in[k]$ with

[TABLE]

the last using the upper bound on $\sigma$ of $(c/\sqrt{m})+(c/\sqrt{n})$ we proved and noting that $n\geq m$.

9.2 MMSB

We sketch the proof of Lemma 7.2 and Theorem 7.2 omitting details, since the proof is somewhat similar to the proof in the LDA case. Let ${\bf D}$ denote a $d\times n$ matrix, where, $D_{ij}=\mbox{Var}(A_{ij}\;|\;{\bf P})$. Clearly we have $D_{ij}=P_{ij}(1-P_{ij})\leq P_{ij}$ and so $\sum_{i}D_{ij}\leq\nu$ for all $j$ and similary for row sums. Also, $\sum_{i,j}D_{ij}\leq\nu(\mbox{Min}(d,n)=\nu d$. Latala’s theorem [18] implies that with high probability,

[TABLE]

which implies $\sigma\leq\nu^{1/4}(d/n)^{1/4}.$ Since $\delta=c\sigma/\sqrt{k}$, to prove (6.12), it suffices to prove that

[TABLE]

which follows since the hypothesis of the Lemma says $n/d$ is a high polynomial in $k/\alpha\varepsilon$ and $|M_{\cdot,\ell}|\geq\nu^{1/8}$. This proves (6.12). The argument for (6.11) is identical to the case of LDA. This proves the Lemma. For the Theorem, we just have to show that upper bound the error guaranteed by Theorem 6.1. satisfies the upper bound claimed here. This is straightforward using the above upper bound on $\sigma$ (and the fact that $\delta=c\sigma/\sqrt{k}$).

10 Proof of Correctness of the Algorithm

In this section we prove the correctness of the algorithm described in Section (6.2) and establish the time complexity.

10.1 Idea of the Proof

The main contribution of the paper is the algorithm, stated formally in Section (6.2) to solve the general problem and the proof of correctness. The algorithm itself is simple. It has $k$ stages; in each stage, it maximizes a carefully chosen linear function $u\cdot x$ over $K^{\prime}$; we prove that the optimum gives us an approximation to one vertex of $K$.

10.1.1 First Step

For the first step, we will pick a random unit vector $u$ in the $k$ dimensional SVD subspace of ${\bf A}$. This subspace is close to the sub-space spanned by $K$. In Stochastic models, the stochastic independence of the data is used to show this (see for example [22]). Here, we have not assumed any stochastic model. Instead, we use a classical theorem called the $\sin\Theta$ theorem [24] from Numerical Analysis. The $\sin\Theta$ theorem helps us prove that the top singular subspace of dimension $k$ of ${\bf A}$ is close to the span of $K$. Now by our Well-Separatedness assumption, for $\ell\not=\ell^{\prime}$, we will see that $M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}$ has length at least poly$(k)\sigma/\sqrt{\delta}$. For a random $u\in K$, the $O(k^{2})$ events that $|u\cdot(M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}})|\geq|M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}|/\text{poly}(k)$ all happen simultaneously by Johnson-Lindenstrauss (and union bound.) This is proved rigorously in Lemma (10.2). [Note that had we picked $u$ uniformly at random from all of ${\bf R}^{d}$, we can only assert that $|u\cdot(M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}})|\geq|M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}|/\sqrt{d}\text{poly}(k)$; the $\sqrt{d}$ factor is not good enough to solve our problem.]

So, if we optimize $u\cdot x$ over $K$, the optimal $x$ is a vertex $M_{\cdot,\ell}$ with $u\cdot M_{\cdot,\ell}$ substantially greater than any other $u\cdot M_{\cdot,\ell^{\prime}}$. But we can only optimize over $K^{\prime}$. Since we make **Proximate  Latent Points **assumption, there is a $\delta$ fraction of $j$ with their $P_{\cdot,j}\approx M_{\cdot,\ell}$, (an assumption formally stated in (6.11)), and so there is a $R\subseteq[n],|R|=\delta n$ with $P_{\cdot,R}\approx M_{\cdot,\ell}$ and so $A_{\cdot,R}\approx M_{\cdot,\ell}$ implying $u\cdot A_{\cdot,R}\approx u\cdot M_{\cdot,\ell}$. Our optimization over $K^{\prime}$ may yield some other subset $R_{1}$ with $u\cdot A_{\cdot,R_{1}}\widetilde{>}u\cdot M_{\cdot,\ell}$. We need to show that whenever any subset 333For reals $a,b$, we say $a\widetilde{>}b$ if $a>b-$(a small number). $R_{1}$ has $u\cdot A_{\cdot,R_{1}}\widetilde{>}u\cdot M_{\cdot,\ell}$, it is also close to $M_{\cdot,\ell}$ in distance. This is in Lemma (10.3). Intuitively, the proof has two parts: We show that any $R_{1}$ with high $u\cdot A_{\cdot,R_{1}}$ also has high $u\cdot P_{\cdot,R_{1}}$. But $P_{\cdot,R_{1}}$ is a convex combination of columns of ${\bf M}$. If the convex combination puts non-trivial weight on vertices other than $M_{\cdot,\ell}$, its dot product with $u$ would go down since we argued above that $u\cdot M_{\cdot,\ell^{\prime}}<<u\cdot M_{\cdot,\ell}$ for $\ell^{\prime}\not=\ell$. So, in the convex combination of the vertices of $K$ yielding $P_{\cdot,R}$ most of the weight must be on $M_{\cdot,\ell}$ and we use this to show that $|P_{\cdot,R}-M_{\cdot,\ell}|$ is small and so also $|A_{\cdot,R}-M_{\cdot,\ell}|$ is small. Thus the optimal $A_{\cdot,R}$ serves as a good approximation to one vertex of $K$.

10.1.2 General Step

In a general step of the algorithm, we have already found $r\leq k$ subsets $R_{1},R_{2},\ldots,R_{r}\subseteq[n],|R_{\ell}|=\delta n$ with $A_{\cdot,R_{\ell}}\approx M_{\cdot,\ell}$ for $\ell=1,2,\ldots,r$ (after some permutation of the indices of $M_{\cdot,\ell}$). We have to ensure that the next stage gets an approximation to a new vertex of $K$. This is non-trivial. Random choice of the next $u$, even constrained to be in the $k-$SVD subspace of ${\bf A}$ need not work: the probability that it works depends on angles of the simplex $K$ and these can be exponentially (in $k$) small. We overcome this problem with a new, but simple idea: Choose a random vector $u$ from a $k-r$ dimensional subspace $W$ obtained by intersecting the SVD $k-$ subspace of ${\bf A}$ with the NULL SPACE of $A_{\cdot,R_{1}},A_{\cdot,R_{2}},\ldots,A_{\cdot,R_{r}}$ and find $\mbox{Max}_{x\in K^{\prime}}|u\cdot x|$. [The absolute value is necessary.]

If (i) $u$ had been in $W^{\prime}=\mbox{Span}({\bf M})\cap\mbox{Null}(M_{\cdot,1},M_{\cdot,2},\ldots,M_{\cdot,r})$, and (ii) we optimized over $K$ instead of $K^{\prime}$, the proof would be easy from well-separateness. Neither is true. Overcoming (i) requires a new Lemma (10.1) which proves that $W,W^{\prime}$ are close in $\sin\Theta$ distance. [The $\sin\Theta$ distance between $W,W^{\prime}$ is $\mbox{Max}_{x\in W}\mbox{Min}_{y\in W^{\prime}}\sin(\angle(x,y))$.] This is a technically involved piece. This in a way extends the Sin-Theta theorem in that it proves that if subspaces $W_{1},W_{2}$ are close in Sin-Theta distance and matrices $\widetilde{M},\widetilde{A}$ are close, then $W_{1}\cap\mbox{Null}(\widetilde{M})$ and $W_{2}\cap\mbox{Null}(\widetilde{A})$ are also close under some conditions (that $\widetilde{A},\widetilde{M}$ are far from singular) which do hold here.

We overcome (ii) in a similar way to what we did for the first step. But, now this is more complicated by the fact that the $M_{\cdot,\ell},M_{\cdot,\ell^{\prime}}$ and $u$ have components along $M_{\cdot,1},M_{\cdot,2},\ldots,M_{\cdot,r}$ as well.

10.2 Proof of the main theorem

We are now ready to prove Theorem 6.2

• Proof.

(of Theorem 6.2): Let

[TABLE]

We next derive an extension of the classical $\sin\Theta$ theorem [24] which could be of general interest but is crucial to the proof of the main theorem. Intuitively, it says that if we take close-by $k$ dim spaces and intersect them with null spaces of close-by matrices, with not-too-small singular values, then the resulting intersections are also close (close in $\sin\Theta$ distance). The reader may consult Section (10.1.2) for the role played by this Lemma in the proof of correctness.

Lemma 10.1

Under the hypothesis of Theorem 6.2, we have:

[TABLE]

• Proof.

For the first assertion, take $x\in U,|x|=1$. We wish to produce a $z\in\mbox{Span}({\bf M})\cap\mbox{Null}(\widetilde{\bf M})$ with $|x-z|\leq\alpha/100k^{4}$. Since $x\in V$, by Lemma 8.2,

[TABLE]

Let, $z=y-\widetilde{\bf M}(\widetilde{\bf M}^{T}\widetilde{\bf M})^{-1}\widetilde{\bf M}^{T}y$ be the component of $y$ in Null$(\widetilde{\bf M})$. [Note: $\widetilde{\bf M}^{T}\widetilde{\bf M}$ is invertible since $s_{r}(\widetilde{\bf M})=\mbox{Min}_{w:|w|=1}|\widetilde{\bf M}w|\geq\mbox{Min}_{x:|x|=1}|{\bf M}x|=s_{k}({\bf M})$ and Claim (8.1).] Since $y\in\mbox{Span}({\bf M})$, $z\in\mbox{Span}({\bf M})$ too.

[TABLE]

since it is a projection operator. We have

[TABLE]

$|x-z|\leq|x-y|+|y-z|$ and using Claim (8.1), the first assertion of the Lemma follows.

To prove (10.18), we argue that Dim$(U)=k-r$ (this plus (8.1) proves (10.18).) $U$ has dimension at least $k-r$. If the dimension of $U$ is greater than $k-r$, then there is an orthonormal set of $k-r+1$ vectors $u_{1},u_{2},\ldots,u_{k-r+1}\in U$. By the first assertion, there are $k-r+1$ vectors $w_{1},w_{2},\ldots,w_{k-r+1}\in\mbox{Span}({\bf M})\cap\mbox{Null}(\widetilde{\bf M})$ with $|w_{t}-u_{t}|\leq\delta_{3},t=1,2,\ldots,k-r+1$. For $t\not=t^{\prime}$, we have

[TABLE]

So the matrix $(w_{1}|w_{2}|\ldots|w_{k-r+1})^{T}\;(w_{1}|w_{2}|\ldots|w_{k-r+1})$ is diagonal-dominant and therefore non-singular. So, $w_{1},w_{2},\ldots,w_{k-r+1}$ are linearly independent vectors in $\mbox{Span}({\bf M})\cap\mbox{Null}(\widetilde{\bf M})$ which contradicts the fact that the dimension of $\mbox{Span}({\bf M})\cap\mbox{Null}(\widetilde{\bf M})$ is $k-r$. This finishes the proof of Lemma  10.1

We have (using  6.13 and Cauchy-Schwartz inequality):

[TABLE]

Claim 10.1

If $\ell,\ell^{\prime}\notin\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$, $\ell\not=\ell^{\prime}$, then,

[TABLE]

• Proof.

$|\mbox{proj}(M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}},Null(\widetilde{\bf M}))$ $=\mbox{Min}_{x}|M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}-\widetilde{\bf M}x|$ $\geq\mbox{Min}_{\beta,x}|M_{\cdot,\ell}-\beta M_{\cdot,\ell^{\prime}}-\widetilde{\bf M}x|$

[TABLE]

[TABLE]

where, the last inequality is from (6.10).

Next, we prove the Lemma that states that $|u\cdot x|$ has an unambiguous optimum over $K$: I..e., there is an $\ell$ so that $|u\cdot M_{\cdot,\ell}|$ is a definite amount higher than any other $|u\cdot M_{\cdot,\ell^{\prime}}|$. The reader may want to consult the intuitive description in Section (10.1.1) for the role played by this Lemma in the proof of correctness. In short, this Lemma would say that if we were able to optimize over $K$, we could get a hold of a new vertex. While this may first seem tautological, the point is that if there were ties for the optimum over $K$, then, instead of a vertex, we may get a point in the interior of a face of $K$. Indeed, since the sides of $K$ are relatively small (compared to $n,d$), it requires some work (this lemma) to rule this out. This alone is not sufficient, since we have access only to $K^{\prime}$, not $K$. The next Lemma will prove that the optimal solutions (not just solution values) over $K$ and $K^{\prime}$ are close.

Lemma 10.2

Let $u$ be as in the algorithm. With probability at least $1-(c/k^{3/2})$, the following hold:

[TABLE]

• Proof.

We can write

[TABLE]

where we use the fact that $q_{\ell}$ can be written as $M_{\cdot,\ell}-\widetilde{\bf M}w^{(\ell)}$ for some $w^{(\ell)}$.

From (6.10), we have ]$|q_{\ell}|\geq\alpha\mbox{Max}_{\ell^{\prime\prime}}|M_{\cdot,\ell^{\prime\prime}}|$. Since $|p_{\ell}|\leq|M_{\cdot,\ell}|$, and $s_{r}(\widetilde{\bf M})=\mbox{Min}_{|x|=1}|\widetilde{\bf M}x|\geq\mbox{Min}_{|y|=1}|{\bf M}y|=s_{k}({\bf M})$, Claim (8.1) implies:

[TABLE]

Recall $u$ in the Theorem statement - $u$ is a random unit length vector in subspace $U$.

[TABLE]

since $u^{T}\widetilde{\bf A}=0.$ So,

[TABLE]

using (10.21) and (10.23). Similarly, for $\ell^{\prime}\not=\ell$. $u\cdot(M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}})=u\cdot\mbox{Proj}(q_{\ell}-q_{\ell^{\prime}},U)+u^{T}\widetilde{\bf M}(w^{(\ell)}-w^{(\ell^{\prime})})$ So, $|u\cdot(M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}})-u\cdot\mbox{Proj}(q_{\ell}-q_{\ell^{\prime}},U)|\leq|u^{T}(\widetilde{\bf M}-\widetilde{\bf A})(w^{(\ell)}-w^{(\ell^{\prime})})|$ (using $u^{T}\widetilde{A}=0$)

[TABLE]

using (10.21) and $|w^{(\ell)}-w^{(\ell^{\prime})}|\leq|\widetilde{\bf M}(w^{(\ell)}-w^{(\ell^{\prime})})|/s_{k}({\bf M})\leq|M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}|/s_{k}({\bf M})$, since, $\widetilde{\bf M}(w^{(\ell)}-w^{(\ell^{\prime})})$ is an orthogonal projection of $M_{\cdot,\ell}-M_{\cdot,\ell^{\prime}}$ into $\mbox{Span}(\widetilde{\bf M})$ (and using Claim  8.1).

Now, $u$ is a random unit length vector in $U$. Now, $\mbox{Proj}(q_{\ell},U),\mbox{Proj}(q_{\ell}-q_{\ell^{\prime}},U),\ell,\ell^{\prime}\in[k]$ are fixed vectors in $U$ (and the choice of $u$ doesn’t dependent on them). Consider the following event ${\cal E}$:

[TABLE]

[TABLE]

The negation of ${\cal E}$ is the union of at most $k^{2}$ events (for each $\ell$ and each $\ell,\ell^{\prime}$) and each of these has a failure probability of at most $1/10k^{3.5}$ (since the $k-1$ volume of $\{x\in U:u\cdot x=0\}$ is at most $\sqrt{k}$ times the volume of the unit ball in $U$). Thus, we have:

[TABLE]

We pay the failure probability and assume from now on that ${\cal E}$ holds.

By (10.18), we have that there is a $q^{\prime}_{\ell}\in U$ with $|q^{\prime}_{\ell}-q_{\ell}|\leq\alpha|q_{\ell}|/(100k^{4})$ which implies (recall $k\geq 2$):

[TABLE]

So, under ${\cal E}$,

[TABLE]

since $|q_{\ell}|\geq|proj({\bf M}_{\cdot,\ell},\mbox{Null}({\bf M}\setminus M_{\cdot,\ell}))|\geq\alpha\mbox{Max}_{\ell^{\prime\prime}}|M_{\cdot,\ell^{\prime\prime}}|$ by (6.10).

By (10.24) and (10.28), $\forall\ell\notin\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$,

[TABLE]

proving the second assertion of the Lemma.

Now we prove the first assertion. For $\ell\notin\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$ and $\ell^{\prime}\notin\{\ell,\ell_{1},\ell_{2},\ldots,\ell_{r}\}$, by (10.25),

[TABLE]

[TABLE]

since, by  10.18, $\exists x\in U,$: $|x-(q_{\ell}-q_{\ell^{\prime}})|\leq\frac{\alpha|q_{\ell}-q_{\ell^{\prime}}|}{100k^{4}}$, we have $|\mbox{Proj}(q_{\ell}-q_{\ell^{\prime}},U)|\geq.99|q_{\ell}-q_{\ell^{\prime}}|\geq.99\alpha\mbox{Max}_{\ell^{\prime\prime}}|M_{\cdot,\ell^{\prime\prime}}|,$ by Claim  10.1. This finishes the proof of the first assertion and of the Lemma.

We just proved that $|u\cdot x|$ has an unambiguous maximum over $K$. The following Lemma shows that if $M_{\cdot,\ell}$ is this optimum, and if $A_{\cdot,S}$ is the optimum of $|u\cdot x|$ over $K^{\prime}$, then, $A_{\cdot,S}\approx M_{\cdot,\ell}$. The idea of the proof is that for the optimal $A_{\cdot,S}$, the corresponding $P_{\cdot,S}$ which is in $K$ is a convex combination of all columns of ${\bf M}$. If the convex combination involves any appreciable amount of non-optimal vertices of $K$, since, by the last Lemma, $|u\cdot x|$ is considerably less at non-optimal vertices than the optimal one, $|u\cdot A_{\cdot,S}|$ would be considerably less than $|u\cdot M_{\cdot,\ell}|$, where, $M_{\cdot,\ell}$ is the optimum over $K$. This produces a contradiction to $A_{\cdot,S}$ being optimal over $K^{\prime}$ since, by (6.11), there is a set $S_{\ell}$ with $|u\cdot A_{\cdot,S_{\ell}}|\approx|u\cdot M_{\cdot,\ell}|$.

Lemma 10.3

Let $R_{r+1}$ be an in algorithm. Define $\ell$ by:

[TABLE]

Then, under the hypothesis of Theorem  6.2 $\ell\notin\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$ and

[TABLE]

• Proof.

Case 1 $u\cdot A_{\cdot,R_{r+1}}\geq 0$.

We scale $u$, so that $|u|=1$ which does not change $R_{r+1}$ found by the algorithm. Now,

[TABLE]

We claim that $\ell\notin\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$. Suppose for contradiction, $\ell\in\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$; wlg, say $\ell=\ell_{1}$. Then, by the hypothesis of Theorem  6.2, we have that $|A_{R_{1}}-M_{\cdot,\ell_{1}}|\leq 150\sigma k^{4}/\alpha\sqrt{\delta}$ and so, $u\cdot M_{\cdot,\ell_{1}}\leq u\cdot A_{\cdot,R_{1}}+(150k^{4}\sigma)/(\alpha\sqrt{\delta})=(150k^{4}\sigma)/(\alpha\sqrt{\delta})$ (since $u\in U$ and so $u\perp A_{\cdot,R_{1}}$). So, for all $\ell^{\prime}$, $u\cdot M_{\cdot,\ell^{\prime}}\leq u\cdot M_{\cdot,\ell_{1}}\leq(150k^{4}\sigma)/(\alpha\sqrt{\delta})$. So, for all $R\subseteq[n]$, $P_{\cdot,R}$ which is in CH$({\bf M})$, satisfies $u\cdot P_{\cdot,R}\leq 150k^{4}\sigma/\alpha\sqrt{\delta}$. So, by Lemma (3.1), $u\cdot A_{\cdot,R_{r+1}}\leq u\cdot P_{\cdot,R_{r+1}}+(\sigma/\sqrt{\delta})\leq((150k^{4}/\alpha)+1)\sigma/\sqrt{\delta}$. But for any $t\notin\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$, we have with $S_{t}$ as in 6.11,

[TABLE]

and so, $u\cdot A_{\cdot,R_{r+1}}$ (which maximizes $u\cdot A_{\cdot,R}$ over all $R,|R|=\delta n$) must be at least $\frac{\alpha\mbox{Max}_{\ell^{\prime\prime}}|M_{\cdot,\ell^{\prime\prime}}|}{11k^{4}}-\frac{5\sigma}{\sqrt{\delta}}$ contradicting $u\cdot A_{\cdot,R_{r+1}}\leq((150k^{4}/\alpha)+1)\sigma/\sqrt{\delta}$ by  6.12. So, $\ell\notin\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$ and by Lemma (10.2),

[TABLE]

We have $|P_{\cdot,j}-M_{\cdot,\ell}|\leq\frac{4\sigma}{\sqrt{\delta}}$ for all $j\in S_{\ell}$, so also $|P_{\cdot,S_{\ell}}-M_{\cdot,\ell}|\leq\frac{4\sigma}{\sqrt{\delta}}$.

[TABLE]

By the definition of $R_{r+1}$,

[TABLE]

For any $\ell^{\prime}\notin\{\ell,\ell_{1},\ell_{2},\ldots,\ell_{r}\}$, we have by Lemma  10.2,

[TABLE]

Also, for $\ell^{\prime}\in\{\ell_{1},\ell_{2},\ldots,\ell_{r}\}$, wlg, say $\ell^{\prime}=\ell_{1}$, we have noting that $|A_{\cdot,R_{1}}-M_{\cdot,\ell_{1}}|\leq 150k^{4}\sigma/(\alpha\sqrt{\delta})$ from the hypothesis of Theorem  6.2:

[TABLE]

Now, $P_{\cdot,R_{r+1}}$ is a convex combination of the columns of ${\bf M}$; say the convex combination is $P_{\cdot,R_{r+1}}={\bf M}w$. From above, we have:

[TABLE]

This and (10.32) imply:

[TABLE]

So, $|P_{\cdot,R_{r+1}}-M_{\cdot,\ell}|=\left|(w_{\ell}-1)M_{\cdot,\ell}+\sum_{\ell^{\prime}\not=\ell}w_{\ell^{\prime}}M_{\cdot,\ell^{\prime}}\right|$

[TABLE]

Now it follows that $|A_{\cdot,R_{r+1}}-M_{\cdot,\ell}|\leq\frac{150k^{4}}{\alpha}\frac{\sigma}{\sqrt{\delta}}$ finishing the proof of the theorem in this case. An exactly symmetric argument proves the theorem in the case when $u\cdot A_{\cdot,S}\leq 0.$

Time Complexity

Our algorithm above is novel in the sense this approach of using successive optimizations to find extreme points of the hidden simplex does not seem to be used in any of the special cases. It also has a more useful consequence: we are able to show that the only way we treat ${\bf A}$ is matrix-vector products and therefore we are able to prove a running time bound of $O^{*}(k\text{ nnz}+k^{2}d)$ on the algorithm. We also use the observation that the SVD at the start can be done in the required time by the classical sub-space power method. The SVD as well as keeping track of the subspace $W$ are done by keeping and updating a $d\times(k-r)$ (possibly dense) matrix whose columns form a basis of $W$. We note that one raw iteration of the standard $k-$means algorithm finds the distance between each of $n$ data points and each of $k$ current cluster centers which takes $O(k\text{nnz})$ time matching the leading term of our total running time.

The first step of the algorithm is to do $O(\ln d)$ subspace-power iterations. Each iteration starts with a $d\times k$ matrix ${\bf Q_{t}}$ with orthonormal columns, multiplies ${\bf A}{\bf A}^{T}{\bf Q_{t}}$ and makes the product’s columns orthonormal by doing a Grahm-Schmidt. The products (first pre-multiply ${\bf Q_{t}}$ by ${\bf A}^{T}$ and then pre-multiply by ${\bf A}$ take time $O(\text{nnz})$. Doing Grahm-Scmidt takes involves dot product of each column with previous columns and subtracting out the component. The columns of ${\bf Q_{t+1}}$ are possibly dense, but, still, each dot product takes time $O(d)$ and there are $k^{2}$ of them for a total of $O(dk^{2})$ per iteration times $O^{*}(1)$ iterations.

The rest of the algorithm has the complexity we claim. We do $k$ rounds in each of which, we must first choose a random $u\in V\cap\mbox{Null}(A_{\cdot,R_{1}},A_{\cdot,R_{2}},\ldots A_{\cdot,R_{r}})$. To do this, we keep a orthonormal basis of $\mbox{Span}(A_{\cdot,R_{1}},A_{\cdot,R_{2}},\ldots A_{\cdot,R_{r}})$; updating this once involves finding the dot product of $A_{\cdot,R_{r+1}}$ with the previous basis in time $O(dk)$, for a total of $dk^{2}$. Now to pick a random $u\in V\cap\mbox{Null}(A_{\cdot,R_{1}},A_{\cdot,R_{2}},\ldots A_{\cdot,R_{r}})$, we just pick a random $u$ from $V$ and then subtract out its component in $\mbox{Span}(A_{\cdot,S_{1}},A_{\cdot,S_{2}},\ldots,A_{\cdot,S_{r}})$. All of this can be done in $O^{*}(k\;\text{nnz}({\bf A})+k^{2}d)$ time. This completes the proof of Theorem 6.2.

11 Conclusion

The dependence of the Well-Separatedness on $k$ could be improved. For Gaussian Mixture Models, one can get $k^{1/4}$, but this is a very special case of our problem. But in any case, something substantially better than $k^{8}$ would seem reasonable to aim for. Another important improvement of the same assumption would be to ask only that each column of ${\bf M}$ be separated in distance (not in perpendicular component) from the others. An empirical study of the speed and quality of solutions of this algorithm in comparison to Algorithms for special cases would be an interesting story of how well asymptotic complexity reflects practical efficacy in this case. The subset-soothing construction should be applicable to other models where there is stochastic Independence, since subset averaging improves variance in general.