Low-degree Lower bounds for clustering in moderate dimension

TL;DR

This paper investigates the computational complexity of clustering Gaussian mixtures in moderate dimensions, establishing new lower bounds and proposing algorithms that match these bounds, thus clarifying the limits of polynomial-time clustering methods.

Contribution

It introduces a novel low-degree polynomial lower bound for clustering in moderate dimensions and presents a non-spectral algorithm that achieves this bound.

Findings

Lower bounds for clustering difficulty in moderate dimensions.

A non-spectral algorithm matching the established lower bounds.

Insights into the transition from spectral to non-parametric regimes.

Abstract

We study the fundamental problem of clustering $n$ points into $K$ groups drawn from a mixture of isotropic Gaussians in $\mathbb{R}^d$. Specifically, we investigate the requisite minimal distance $\Delta$ between mean vectors to partially recover the underlying partition. While the minimax-optimal threshold for $\Delta$ is well-established, a significant gap exists between this information-theoretic limit and the performance of known polynomial-time procedures. Although this gap was recently characterized in the high-dimensional regime ($n \leq dK$), it remains largely unexplored in the moderate-dimensional regime ($n \geq dK$). In this manuscript, we address this regime by establishing a new low-degree polynomial lower bound for the moderate-dimensional case when $d \geq K$. We show that while the difficulty of clustering for $n \leq dK$ is primarily driven by dimension reduction and spectral methods, the moderate-dimensional regime involves more delicate phenomena leading to a "non-parametric rate". We provide a novel non-spectral algorithm matching this rate, shedding new light on the computational limits of the clustering problem in moderate dimension.