From A-to-Z Review of Clustering Validation Indices

TL;DR

This paper provides a comprehensive review of clustering validation indices, categorizing their mathematical foundations, evaluating their performance on common algorithms, and proposing a framework to guide their selection for different applications.

Contribution

It offers a detailed categorization and analysis of internal and external clustering validation indices, along with performance evaluation and a classification framework for better selection.

Findings

Performance varies across different validation indices and clustering algorithms.

A classification framework helps in selecting suitable validation measures.

Evaluation on common algorithms like ECA* demonstrates practical applicability.

Abstract

Data clustering involves identifying latent similarities within a dataset and organizing them into clusters or groups. The outcomes of various clustering algorithms differ as they are susceptible to the intrinsic characteristics of the original dataset, including noise and dimensionality. The effectiveness of such clustering procedures directly impacts the homogeneity of clusters, underscoring the significance of evaluating algorithmic outcomes. Consequently, the assessment of clustering quality presents a significant and complex endeavor. A pivotal aspect affecting clustering validation is the cluster validity metric, which aids in determining the optimal number of clusters. The main goal of this study is to comprehensively review and explain the mathematical operation of internal and external cluster validity indices, but not all, to categorize these indices and to brainstorm suggestions for future advancement of clustering validation research. In addition, we review and evaluate the performance of internal and external clustering validation indices on the most common clustering algorithms, such as the evolutionary clustering algorithm star (ECA*). Finally, we suggest a classification framework for examining the functionality of both internal and external clustering validation measures regarding their ideal values, user-friendliness, responsiveness to input data, and appropriateness across various fields. This classification aids researchers in selecting the appropriate clustering validation measure to suit their specific requirements.