A correlation-based fuzzy cluster validity index with secondary options detector

TL;DR

This paper introduces the WP index, a new correlation-based fuzzy cluster validity index that effectively determines the optimal number of clusters and secondary options across various datasets, outperforming existing indexes.

Contribution

The paper proposes the WP index, a novel correlation-based fuzzy cluster validity index that improves detection of the optimal and secondary cluster numbers, especially with high fuzziness.

Findings

WP index outperforms existing indexes in accuracy

Effective across artificial, real-world, and image datasets

Remains robust with large fuzziness parameter m

Abstract

The optimal number of clusters is one of the main concerns when applying cluster analysis. Several cluster validity indexes have been introduced to address this problem. However, in some situations, there is more than one option that can be chosen as the final number of clusters. This aspect has been overlooked by most of the existing works in this area. In this study, we introduce a correlation-based fuzzy cluster validity index known as the Wiroonsri-Preedasawakul (WP) index. This index is defined based on the correlation between the actual distance between a pair of data points and the distance between adjusted centroids with respect to that pair. We evaluate and compare the performance of our index with several existing indexes, including Xie-Beni, Pakhira-Bandyopadhyay-Maulik, Tang, Wu-Li, generalized C, and Kwon2. We conduct this evaluation on four types of datasets: artificial datasets, real-world datasets, simulated datasets with ranks, and image datasets, using the fuzzy c-means algorithm. Overall, the WP index outperforms most, if not all, of these indexes in terms of accurately detecting the optimal number of clusters and providing accurate secondary options. Moreover, our index remains effective even when the fuzziness parameter $m$ is set to a large value. Our R package called UniversalCVI used in this work is available at https://CRAN.R-project.org/package=UniversalCVI.