A new approach for evaluating internal cluster validation indices

TL;DR

This paper reviews existing internal cluster validation indices and proposes a new evaluation approach to better assess clustering quality without external information.

Contribution

It introduces a novel method for evaluating internal validation indices, addressing limitations of previous approaches.

Findings

Analysis of existing evaluation methods

Introduction of a new evaluation approach

Improved assessment of clustering quality

Abstract

A vast number of different methods are available for unsupervised classification. Since no algorithm and parameter setting performs best in all types of data, there is a need for cluster validation to select the actually best-performing algorithm. Several indices were proposed for this purpose without using any additional (external) information. These internal validation indices can be evaluated by applying them to classifications of datasets with a known cluster structure. Evaluation approaches differ in how they use the information on the ground-truth classification. This paper reviews these approaches, considering their advantages and disadvantages, and then suggests a new approach.