(Un)detectable cluster structure in sparse networks

TL;DR

This paper investigates the detectability of cluster structures in sparse networks, revealing a critical separation threshold below which clusters cannot be recovered regardless of network size, with implications for graph clustering and data mining.

Contribution

It provides an analytic characterization of the phase transition in cluster recoverability in sparse networks, including formulas for the critical separation based on degree distribution.

Findings

Identifies a sharp transition from un-recoverable to recoverable clusters.

Derives formulas for the critical separation threshold.

Shows that some cluster structures are undetectable even in infinitely large networks.

Abstract

We study the problem of recovering a known cluster structure in a sparse network, also known as the planted partitioning problem, by means of statistical mechanics. We find a sharp transition from un-recoverable to recoverable structure as a function of the separation of the clusters. For multivariate data, such transitions have been observed frequently, but always as a function of the number of data points provided, i.e. given a large enough data set, two point clouds can always be recognized as different clusters, as long as their separation is non-zero. In contrast, for the sparse networks studied here, a cluster structure remains undetectable even in an infinitely large network if a critical separation is not exceeded. We give analytic formulas for this critical separation as a function of the degree distribution of the network and calculate the shape of the recoverability-transition. Our findings have implications for unsupervised learning and data-mining in relational data bases and provide bounds on the achievable performance of graph clustering algorithms.