Algorithms for finding $k$ in $k$-means

TL;DR

This paper introduces a data-driven method to determine the number of clusters in $k$-means clustering, providing the first polynomial-time algorithm under natural assumptions, applicable to Gaussian and sub-Gaussian mixtures.

Contribution

It proposes a novel, deterministic framework with assumptions like GT clustering and NTSC, enabling the identification of $k$ without prior knowledge, and offers a polynomial-time algorithm for this task.

Findings

Algorithm successfully identifies $k$ in synthetic and real data.

Applicable to Gaussian and sub-Gaussian mixture models.

Provides theoretical guarantees under natural assumptions.

Abstract

$k-$means Clustering requires as input the exact value of $k$, the number of clusters. Two challenges are open: (i) Is there a data-determined definition of $k$ which is provably correct and (ii) Is there a polynomial time algorithm to find $k$ from data ? This paper provides the first affirmative answers to both these questions. As common in the literature, we assume that the data admits an unknown Ground Truth (GT) clustering with cluster centers separated. This assumption alone is not sufficient to answer Yes to (i). We assume a novel, but natural second constraint called no tight sub-cluster (NTSC) which stipulates that no substantially large subset of a GT cluster can be "tighter" (in a sense we define) than the cluster. Our yes answer to (i) and (ii) are under these two deterministic assumptions. We also give polynomial time algorithm to identify $k$. Our algorithm relies on NTSC to peel off one cluster at a time by identifying points which are tightly packed. We are also able to show that our algorithm(s) apply to data generated by mixtures of Gaussians and more generally to mixtures of sub-Gaussian pdf's and hence are able to find the number of components of the mixture from data. To our knowledge, previous results for these specialized settings as well, assume generally that $k$ is given besides the data.