Sanity Check for External Clustering Validation Benchmarks using Internal Validation Measures

TL;DR

This paper introduces a new method to evaluate the alignment between class labels and true clusters across datasets, improving the reliability of external clustering validation benchmarks.

Contribution

It proposes a novel framework for extending internal validation measures to compare cluster-label matching across datasets, addressing a key gap in external validation.

Findings

The generalized Calinski-Harabasz index effectively evaluates CLM across datasets.

The proposed measures satisfy four new axioms for between-dataset internal validation.

Evaluating CLM before external validation improves benchmarking reliability.

Abstract

We address the lack of reliability in benchmarking clustering techniques based on labeled datasets. A standard scheme in external clustering validation is to use class labels as ground truth clusters, based on the assumption that each class forms a single, clearly separated cluster. However, as such cluster-label matching (CLM) assumption often breaks, the lack of conducting a sanity check for the CLM of benchmark datasets casts doubt on the validity of external validations. Still, evaluating the degree of CLM is challenging. For example, internal clustering validation measures can be used to quantify CLM within the same dataset to evaluate its different clusterings but are not designed to compare clusterings of different datasets. In this work, we propose a principled way to generate between-dataset internal measures that enable the comparison of CLM across datasets. We first determine four axioms for between-dataset internal measures, complementing Ackerman and Ben-David's within-dataset axioms. We then propose processes to generalize internal measures to fulfill these new axioms, and use them to extend the widely used Calinski-Harabasz index for between-dataset CLM evaluation. Through quantitative experiments, we (1) verify the validity and necessity of the generalization processes and (2) show that the proposed between-dataset Calinski-Harabasz index accurately evaluates CLM across datasets. Finally, we demonstrate the importance of evaluating CLM of benchmark datasets before conducting external validation.