Adaptive cut reveals multiscale complexity in networks

TL;DR

This paper introduces the adaptive cut, a multi-level dendrogram cutting method optimized via MCMC with simulated annealing, improving clustering quality in network analysis over traditional single-level cuts.

Contribution

The paper presents the adaptive cut, a novel multi-level dendrogram cutting technique that enhances clustering accuracy and introduces the balancedness score to predict multi-level cut benefits.

Findings

Significant improvements in modularity and partition density over single cuts.

Effective across various clustering methods and datasets.

Introduces the balancedness score for dendrogram evaluation.

Abstract

Hierarchical clustering and community detection are important problems in machine learning and complex network analysis. A common approach to identify clusters is to simply cut dendrograms at some threshold. However, single-level cuts are often suboptimal in terms of capturing underlying structure in the data, especially when the dendrogram is unbalanced. In this paper, we present the adaptive cut, a novel method that leverages the hierarchical structure of dendrograms by employing multi-level cuts to overcome the limitations of single-level approaches. The adaptive cut optimizes an objective function using a Markov chain Monte Carlo with simulated annealing, resulting in better partitions. We demonstrate the effectiveness of the adaptive cut through applications to link clustering and modularity optimization, but note that the method is applicable to any clustering task that relies on a dendrogram and an objective function. Beyond the adaptive cut, we introduce the balancedness score, an information-theoretic metric that quantifies how balanced a dendrogram is. Balancedness predicts the potential benefits of using multi-level cuts. For the community detection examples, we evaluate our method on more than 200 real-world networks and multiple synthetic datasets, demonstrating significant improvements in partition density and modularity over traditional single-cut approaches. In addition, we show the generality of the adaptive cut by applying it across various hierarchical clustering techniques and objective functions. Our results indicate that the adaptive cut provides a robust and versatile tool for improving clustering outcomes.