Clustering by Direct Optimization of the Medoid Silhouette

TL;DR

This paper introduces efficient methods for directly optimizing the medoid-based Silhouette measure in clustering, significantly speeding up the process while maintaining accuracy, especially for large datasets.

Contribution

It presents two fast algorithms for direct optimization of the medoid Silhouette, combining ideas from PAM and FasterPAM, with theoretical analysis and substantial speed improvements.

Findings

Achieved a 10464× speedup on real data with 30,000 samples and 100 clusters.

Provided algorithms that match the original Silhouette results with faster computation.

Demonstrated the effectiveness of the methods on large-scale clustering tasks.

Abstract

The evaluation of clustering results is difficult, highly dependent on the evaluated data set and the perspective of the beholder. There are many different clustering quality measures, which try to provide a general measure to validate clustering results. A very popular measure is the Silhouette. We discuss the efficient medoid-based variant of the Silhouette, perform a theoretical analysis of its properties, and provide two fast versions for the direct optimization. We combine ideas from the original Silhouette with the well-known PAM algorithm and its latest improvements FasterPAM. One of the versions guarantees equal results to the original variant and provides a run speedup of $O(k^2)$. In experiments on real data with 30000 samples and $k$=100, we observed a 10464$\times$ speedup compared to the original PAMMEDSIL algorithm.