A High-Performance External Validity Index for Clustering with a Large Number of Clusters

TL;DR

This paper presents SMBP, a scalable and efficient external validity index for clustering with many clusters, leveraging stable matching to reduce computational complexity while maintaining accuracy.

Contribution

Introduces SMBP, a novel stable matching-based clustering evaluation method that significantly improves computational efficiency for large-scale datasets.

Findings

SMBP achieves comparable accuracy to MWM.

SMBP reduces computational complexity to O(N^2).

SMBP is effective on large and unbalanced datasets.

Abstract

This paper introduces the Stable Matching Based Pairing (SMBP) algorithm, a high-performance external validity index for clustering evaluation in large-scale datasets with a large number of clusters. SMBP leverages the stable matching framework to pair clusters across different clustering methods, significantly reducing computational complexity to $O(N^2)$, compared to traditional Maximum Weighted Matching (MWM) with $O(N^3)$ complexity. Through comprehensive evaluations on real-world and synthetic datasets, SMBP demonstrates comparable accuracy to MWM and superior computational efficiency. It is particularly effective for balanced, unbalanced, and large-scale datasets with a large number of clusters, making it a scalable and practical solution for modern clustering tasks. Additionally, SMBP is easily implementable within machine learning frameworks like PyTorch and TensorFlow, offering a robust tool for big data applications. The algorithm is validated through extensive experiments, showcasing its potential as a powerful alternative to existing methods such as Maximum Match Measure (MMM) and Centroid Ratio (CR).