Clustering with the Average Silhouette Width

TL;DR

This paper investigates using the Average Silhouette Width as an objective function for clustering, proposing new algorithms and evaluating their performance through simulations and real data analysis.

Contribution

It introduces two algorithms, OSil and FOSil, for optimizing the ASW as a clustering objective, and assesses their effectiveness compared to existing methods.

Findings

The new methods are useful and sensible in many cases.

ASW satisfies some axioms for cluster quality functions.

Weaknesses in using ASW for estimating the number of clusters are identified.

Abstract

The Average Silhouette Width (ASW; Rousseeuw (1987)) is a popular cluster validation index to estimate the number of clusters. Here we address the question whether it also is suitable as a general objective function to be optimized for finding a clustering. We will propose two algorithms (the standard version OSil and a fast version FOSil) and compare them with existing clustering methods in an extensive simulation study covering the cases of a known and unknown number of clusters. Real data sets are also analysed, partly exploring the use of the new methods with non-Euclidean distances. We will also show that the ASW satisfies some axioms that have been proposed for cluster quality functions (Ackerman and Ben-David (2009)). The new methods prove useful and sensible in many cases, but some weaknesses are also highlighted. These also concern the use of the ASW for estimating the number of clusters together with other methods, which is of general interest due to the popularity of the ASW for this task.