Cluster Analysis via Random Partition Distributions

TL;DR

This paper introduces CaviarPD, a novel stochastic clustering method based on random partition distributions, offering competitive results and uncertainty visualization without subjective linkage choices.

Contribution

The paper presents CaviarPD, a new stochastic clustering approach that leverages random partition distributions and expected loss minimization, differing from traditional deterministic methods.

Findings

CaviarPD produces clustering results comparable to hierarchical and k-medoids methods.

It eliminates the need for subjective linkage method choices.

Provides an intuitive graphical representation of clustering uncertainty.

Abstract

Hierarchical and k-medoids clustering are deterministic clustering algorithms based on pairwise distances. Using these same pairwise distances, we propose a novel stochastic clustering method based on random partition distributions. We call our method CaviarPD, for cluster analysis via random partition distributions. CaviarPD first samples clusterings from a random partition distribution and then finds the best cluster estimate based on these samples using algorithms to minimize an expected loss. We compare CaviarPD with hierarchical and k-medoids clustering through eight case studies. Cluster estimates based on our method are competitive with those of hierarchical and k-medoids clustering. They also do not require the subjective choice of the linkage method necessary for hierarchical clustering. Furthermore, our distribution-based procedure provides an intuitive graphical representation to assess clustering uncertainty.