The elbow statistic: Multiscale clustering statistical significance

TL;DR

ElbowSig is a statistical framework for assessing the significance of clustering structures across multiple resolutions, enabling multiscale inference and improving detection of meaningful data organization.

Contribution

It formalizes the elbow heuristic into a hypothesis testing framework applicable to various clustering methods, addressing multiscale clustering significance.

Findings

Controls Type-I error on unstructured data

Detects multiscale data organization effectively

Compatible with diverse clustering algorithms

Abstract

Selecting the number of clusters remains a fundamental challenge in unsupervised learning. Existing approaches typically focus on identifying a single "optimal" partition, often overlooking statistically meaningful structure present across multiple resolutions. We introduce ElbowSig, a general inferential framework for assessing clustering structure over a range of resolutions. The method formalizes the elbow heuristic by defining a normalized discrete curvature statistic based on the sequence of within-cluster heterogeneity values, and evaluates its significance relative to a null distribution of unstructured data. This yields hypothesis tests across resolutions, enabling simultaneous inference at multiple clustering scales. We derive the asymptotic behavior of the null statistic in both large-sample and high-dimensional regimes, characterizing its limiting form and variability. Because it depends only on the heterogeneity sequence, ElbowSig is compatible with a wide range of clustering algorithms, including hard, fuzzy, and model-based methods. Experiments on synthetic and real datasets show that the procedure controls Type-I error under unstructured data while providing power to detect multiscale organization, revealing structure that is often missed by single-resolution selection criteria.