Rethinking Divisive Hierarchical Clustering from a Distributional Perspective

TL;DR

This paper proposes a novel distributional kernel approach for Divisive Hierarchical Clustering, addressing shortcomings of set-oriented criteria and improving dendrogram quality for biological data analysis.

Contribution

It introduces a distributional kernel-based criterion for DHC, ensuring more accurate and meaningful clustering aligned with ground-truth and biological regions.

Findings

The method guarantees a lower bound of total similarity of all clusters.

It outperforms existing methods on artificial datasets.

It accurately captures biological regions in Spatial Transcriptomics data.

Abstract

We uncover that current objective-based Divisive Hierarchical Clustering (DHC) methods produce a dendrogram that does not have three desired properties i.e., no unwarranted splitting, group similar clusters into a same subset, ground-truth correspondence. This shortcoming has their root cause in using a set-oriented bisecting assessment criterion. We show that this shortcoming can be addressed by using a distributional kernel, instead of the set-oriented criterion; and the resultant clusters achieve a new distribution-oriented objective to maximize the total similarity of all clusters (TSC). Our theoretical analysis shows that the resultant dendrogram guarantees a lower bound of TSC. The empirical evaluation shows the effectiveness of our proposed method on artificial and Spatial Transcriptomics (bioinformatics) datasets. Our proposed method successfully creates a dendrogram that is consistent with the biological regions in a Spatial Transcriptomics dataset, whereas other contenders fail.