Phase transitions and optimal algorithms in high-dimensional Gaussian mixture clustering

TL;DR

This paper analyzes phase transitions in high-dimensional Gaussian mixture clustering, identifying thresholds for cluster detectability and the performance limits of optimal algorithms using statistical physics methods.

Contribution

It determines the critical data density and cluster separation thresholds for successful clustering, revealing gaps between theoretical and algorithmic limits in high dimensions.

Findings

Identifies the information-theoretic threshold for cluster recovery.

Determines the Bayes-optimal accuracy in high-dimensional clustering.

Shows a gap between optimal and known algorithm performance when clusters are numerous.

Abstract

We consider the problem of Gaussian mixture clustering in the high-dimensional limit where the data consists of $m$ points in $n$ dimensions, $n,m \rightarrow \infty$ and $\alpha = m/n$ stays finite. Using exact but non-rigorous methods from statistical physics, we determine the critical value of $\alpha$ and the distance between the clusters at which it becomes information-theoretically possible to reconstruct the membership into clusters better than chance. We also determine the accuracy achievable by the Bayes-optimal estimation algorithm. In particular, we find that when the number of clusters is sufficiently large, $r > 4 + 2 \sqrt{\alpha}$, there is a gap between the threshold for information-theoretically optimal performance and the threshold at which known algorithms succeed.