Simultaneous Estimation of Number of Clusters and Feature Sparsity in Clustering High-Dimensional Data

TL;DR

This paper introduces a resampling-based method for simultaneously estimating the number of clusters and feature sparsity in high-dimensional data, improving clustering accuracy and interpretability.

Contribution

It proposes a novel resampling approach within the sparse K-means framework to jointly estimate cluster number and feature sparsity, addressing a gap in exploratory clustering methods.

Findings

Outperforms classical methods in low-dimensional data for estimating K.

Shows superior performance in high-dimensional data for estimating K and feature sparsity.

Achieves better clustering accuracy with fewer predictive genes in real datasets.

Abstract

Estimating the number of clusters (K) is a critical and often difficult task in cluster analysis. Many methods have been proposed to estimate K, including some top performers using resampling approach. When performing cluster analysis in high-dimensional data, simultaneous clustering and feature selection is needed for improved interpretation and performance. To our knowledge, none has investigated simultaneous estimation of K and feature selection in an exploratory cluster analysis. In this paper, we propose a resampling method to meet this gap and evaluate its performance under the sparse K-means clustering framework. The proposed target function balances between sensitivity and specificity of clustering evaluation of pairwise subjects from clustering of full and subsampled data. Through extensive simulations, the method performs among the best over classical methods in estimating K in low-dimensional data. For high-dimensional simulation data, it also shows superior performance to simultaneously estimate K and feature sparsity parameter. Finally, we evaluated the methods in four microarray, two RNA-seq, one SNP and two non-omics datasets. The proposed method achieves better clustering accuracy with fewer selected predictive genes in almost all real applications.