Clustering with Spectral Norm and the k-means Algorithm

TL;DR

This paper introduces a simple, model-agnostic clustering algorithm based on spectral norms and the k-means method, which effectively clusters data under a proximity condition without relying on probabilistic generative assumptions.

Contribution

The authors establish a proximity condition that guarantees clustering success with k-means without generative assumptions, and prove convergence of k-means with spurious points.

Findings

The proximity condition is satisfied in common generative models.

The k-means algorithm converges to true centers despite spurious points.

New techniques for enhancing inter-center separation to standard deviation ratio.

Abstract

There has been much progress on efficient algorithms for clustering data points generated by a mixture of $k$ probability distributions under the assumption that the means of the distributions are well-separated, i.e., the distance between the means of any two distributions is at least $\Omega(k)$ standard deviations. These results generally make heavy use of the generative model and particular properties of the distributions. In this paper, we show that a simple clustering algorithm works without assuming any generative (probabilistic) model. Our only assumption is what we call a "proximity condition": the projection of any data point onto the line joining its cluster center to any other cluster center is $\Omega(k)$ standard deviations closer to its own center than the other center. Here the notion of standard deviations is based on the spectral norm of the matrix whose rows represent the difference between a point and the mean of the cluster to which it belongs. We show that in the generative models studied, our proximity condition is satisfied and so we are able to derive most known results for generative models as corollaries of our main result. We also prove some new results for generative models - e.g., we can cluster all but a small fraction of points only assuming a bound on the variance. Our algorithm relies on the well known $k$-means algorithm, and along the way, we prove a result of independent interest -- that the $k$-means algorithm converges to the "true centers" even in the presence of spurious points provided the initial (estimated) centers are close enough to the corresponding actual centers and all but a small fraction of the points satisfy the proximity condition. Finally, we present a new technique for boosting the ratio of inter-center separation to standard deviation.