Interpretable Clustering with the Distinguishability Criterion

TL;DR

This paper introduces the Distinguishability criterion, a new global measure for validating and determining the optimal number of clusters in various clustering methods, supported by simulations and real data.

Contribution

The paper proposes the Distinguishability criterion and a unified computational framework that integrates it with existing clustering algorithms for improved validation.

Findings

Effective validation of cluster separability

Compatibility with multiple clustering methods

Successful application to real and simulated data

Abstract

Cluster analysis is a popular unsupervised learning tool used in many disciplines to identify heterogeneous sub-populations within a sample. However, validating cluster analysis results and determining the number of clusters in a data set remains an outstanding problem. In this work, we present a global criterion called the Distinguishability criterion to quantify the separability of identified clusters and validate inferred cluster configurations. Our computational implementation of the Distinguishability criterion corresponds to the Bayes risk of a randomized classifier under the 0-1 loss. We propose a combined loss function-based computational framework that integrates the Distinguishability criterion with many commonly used clustering procedures, such as hierarchical clustering, k-means, and finite mixture models. We present these new algorithms as well as the results from comprehensive data analysis based on simulation studies and real data applications.