Hierarchical Agglomerative Graph Clustering in Poly-Logarithmic Depth

TL;DR

This paper introduces ParHAC, a parallel hierarchical agglomerative clustering algorithm with sublinear depth that scales efficiently to large datasets, providing near-optimal clustering quality and significant speedups over traditional methods.

Contribution

The paper presents the first efficient parallel HAC algorithm with poly-logarithmic depth and provides a $(1+ ext{epsilon})$-approximation for average-linkage clustering on large graphs.

Findings

Achieves a 50.1x speedup over the best sequential baseline.

Can cluster a 124-billion-edge graph in just over three hours.

Maintains clustering quality comparable to exact HAC.

Abstract

Obtaining scalable algorithms for hierarchical agglomerative clustering (HAC) is of significant interest due to the massive size of real-world datasets. At the same time, efficiently parallelizing HAC is difficult due to the seemingly sequential nature of the algorithm. In this paper, we address this issue and present ParHAC, the first efficient parallel HAC algorithm with sublinear depth for the widely-used average-linkage function. In particular, we provide a $(1+\epsilon)$-approximation algorithm for this problem on $m$ edge graphs using $\tilde{O}(m)$ work and poly-logarithmic depth. Moreover, we show that obtaining similar bounds for exact average-linkage HAC is not possible under standard complexity-theoretic assumptions. We complement our theoretical results with a comprehensive study of the ParHAC algorithm in terms of its scalability, performance, and quality, and compare with several state-of-the-art sequential and parallel baselines. On a broad set of large publicly-available real-world datasets, we find that ParHAC obtains a 50.1x speedup on average over the best sequential baseline, while achieving quality similar to the exact HAC algorithm. We also show that ParHAC can cluster one of the largest publicly available graph datasets with 124 billion edges in a little over three hours using a commodity multicore machine.