A Simple Algorithm for Clustering Discrete Distributions

TL;DR

This paper introduces a simple, rotation-invariant clustering algorithm for discrete Bernoulli distributions and continuous distributions like Gaussians, based on projections and low-rank approximations.

Contribution

The paper presents a novel, geometric clustering algorithm that is invariant to rotations and applies to both discrete and continuous distributions, confirming a conjecture by McSherry.

Findings

Algorithm succeeds under a natural separation condition

Works for both Bernoulli and high-dimensional Gaussian distributions

Provides a unified geometric approach for different distribution types

Abstract

We propose a simple, projection-based algorithm for clustering mixtures of discrete (Bernoulli) distributions. Unlike previous approaches that rely on coordinate-specific ``combinatorial projections,'' our algorithm is rotationally invariant and works by projecting samples onto approximate centers obtained via a $k$-means computation on the best rank-$k$ approximation of the data matrix. This resolves a conjecture of McSherry on the existence of such geometric algorithms for discrete distributions. The same algorithm also applies to continuous distributions such as high-dimensional Gaussians, providing a unified approach across distribution types. We prove that the algorithm succeeds under a natural separation condition on the cluster centers.