Optimal Parallel Algorithms for Dendrogram Computation and Single-Linkage Clustering

TL;DR

This paper introduces new parallel algorithms for efficiently computing Single-Linkage Dendrograms, significantly improving speed over traditional methods, and enabling handling of billion-scale trees in practical applications.

Contribution

The paper presents novel parallel algorithms for SLD computation with theoretical guarantees and practical efficiency, surpassing existing work in speed and scalability.

Findings

Achieved up to 150x speedup over Union-Find algorithms.

Developed algorithms with work complexity O(n log h) and depth complexity O(log^2 n log^2 h).

Enabled fast processing of billion-scale trees.

Abstract

Computing a Single-Linkage Dendrogram (SLD) is a key step in the classic single-linkage hierarchical clustering algorithm. Given an input edge-weighted tree $T$, the SLD of $T$ is a binary dendrogram that summarizes the $n-1$ clusterings obtained by contracting the edges of $T$ in order of weight. Existing algorithms for computing the SLD all require $\Omega(n\log n)$ work where $n = |T|$. Furthermore, to the best of our knowledge no prior work provides a parallel algorithm obtaining non-trivial speedup for this problem. In this paper, we design faster parallel algorithms for computing SLDs both in theory and in practice based on new structural results about SLDs. In particular, we obtain a deterministic output-sensitive parallel algorithm based on parallel tree contraction that requires $O(n \log h)$ work and $O(\log^2 n \log^2 h)$ depth, where $h$ is the height of the output SLD. We also give a deterministic bottom-up algorithm for the problem inspired by the nearest-neighbor chain algorithm for hierarchical agglomerative clustering, and show that it achieves $O(n\log h)$ work and $O(h \log n)$ depth. Our results are based on a novel divide-and-conquer framework for building SLDs, inspired by divide-and-conquer algorithms for Cartesian trees. Our new algorithms can quickly compute the SLD on billion-scale trees, and obtain up to 150x speedup over the highly-efficient Union-Find algorithm typically used to compute SLDs in practice.