CoreSPECT: Enhancing Clustering Algorithms via an Interplay of Density and Geometry

TL;DR

CoreSPECT introduces a novel framework that enhances clustering algorithms by exploiting the density-geometry correlation in data, leading to significant improvements in clustering performance and efficiency across diverse datasets.

Contribution

The paper presents CoreSPECT, a new framework that improves clustering algorithms by leveraging density-geometry correlations and multi-layer propagation, with theoretical support and extensive experiments.

Findings

Improves K-Means NMI by 20% on average.

Boosts HDBSCAN NMI by over 100% on average.

Achieves competitive or superior results to state-of-the-art methods.

Abstract

In this paper, we provide a novel perspective on the underlying structure of real-world data with ground-truth clusters via characterization of an abundantly observed yet often overlooked density-geometry correlation, that manifests itself as a multi-layered manifold structure. We leverage this correlation to design CoreSPECT (Core Space Projection based Enhancement of Clustering Techniques), a general framework that improves the performance of generic clustering algorithms. Our framework boosts the performance of clustering algorithms by applying them to strategically selected regions, then extending the partial partition to a complete partition for the dataset using a novel neighborhood graph based multi-layer propagation procedure. We provide initial theoretical support of the functionality of our framework under the assumption of our model, and then provide large-scale real-world experiments on 19 datasets that include standard image datasets as well as genomics datasets. We observe two notable improvements. First, CoreSPECT improves the NMI of K-Means by 20% on average, making it competitive to (and in some cases surpassing) the state-of-the-art manifold-based clustering algorithms, while being orders of magnitude faster. Secondly, our framework boosts the NMI of HDBSCAN by more than 100% on average, making it competitive to the state-of-the-art in several cases without requiring the true number of clusters and hyper-parameter tuning. The overall ARI improvements are higher.