Parallel Hierarchical Agglomerative Clustering in Low Dimensions

TL;DR

This paper develops efficient parallel algorithms for approximate hierarchical clustering with non-monotone linkage functions like centroid and Ward's in low dimensions, and proves hardness results in high dimensions.

Contribution

It introduces NC algorithms for low-dimensional HAC with non-monotone linkages and establishes complexity bounds and hardness results across dimensions.

Findings

NC algorithms for low-dimensional HAC with non-monotone linkages

Hierarchy height is poly(log n) for constant-approximate HAC in low dimensions

HAC with these linkages is CC-hard in arbitrary dimensions

Abstract

Hierarchical Agglomerative Clustering (HAC) is an extensively studied and widely used method for hierarchical clustering in $\mathbb{R}^k$ based on repeatedly merging the closest pair of clusters according to an input linkage function $d$. Highly parallel (i.e., NC) algorithms are known for $(1+\epsilon)$-approximate HAC (where near-minimum rather than minimum pairs are merged) for certain linkage functions that monotonically increase as merges are performed. However, no such algorithms are known for many important but non-monotone linkage functions such as centroid and Ward's linkage. In this work, we show that a general class of non-monotone linkage functions -- which include centroid and Ward's distance -- admit efficient NC algorithms for $(1+\epsilon)$-approximate HAC in low dimensions. Our algorithms are based on a structural result which may be of independent interest: the height of the hierarchy resulting from any constant-approximate HAC on $n$ points for this class of linkage functions is at most $\operatorname{poly}(\log n)$ as long as $k = O(\log \log n / \log \log \log n)$. Complementing our upper bounds, we show that NC algorithms for HAC with these linkage functions in \emph{arbitrary} dimensions are unlikely to exist by showing that HAC is CC-hard when $d$ is centroid distance and $k = n$.