When Does Bottom-up Beat Top-down in Hierarchical Community Detection?

TL;DR

This paper demonstrates that bottom-up hierarchical clustering algorithms can recover community structures more effectively than top-down methods, especially at intermediate levels, supported by theoretical guarantees and empirical evidence.

Contribution

It provides the first theoretical guarantees for bottom-up algorithms in hierarchical community detection and shows they outperform top-down methods in certain regimes.

Findings

Bottom-up algorithms achieve exact recovery at intermediate hierarchy levels.

They have less restrictive conditions for community recovery than top-down algorithms.

Empirical results confirm the superiority of bottom-up methods on synthetic and real data.

Abstract

Hierarchical clustering of networks consists in finding a tree of communities, such that lower levels of the hierarchy reveal finer-grained community structures. There are two main classes of algorithms tackling this problem. Divisive (top-down) algorithms recursively partition the nodes into two communities, until a stopping rule indicates that no further split is needed. In contrast, agglomerative (bottom-up) algorithms first identify the smallest community structure and then repeatedly merge the communities using a linkage method. In this article, we establish theoretical guarantees for the recovery of the hierarchical tree and community structure of a Hierarchical Stochastic Block Model by a bottom-up algorithm. We also establish that this bottom-up algorithm attains the information-theoretic threshold for exact recovery at intermediate levels of the hierarchy. Notably, these recovery conditions are less restrictive compared to those existing for top-down algorithms. This shows that bottom-up algorithms extend the feasible region for achieving exact recovery at intermediate levels. Numerical experiments on both synthetic and real data sets confirm the superiority of bottom-up algorithms over top-down algorithms. We also observe that top-down algorithms can produce dendrograms with inversions. These findings contribute to a better understanding of hierarchical clustering techniques and their applications in network analysis.