Spectral redemption: clustering sparse networks

TL;DR

This paper introduces a new spectral clustering method based on non-backtracking walks that effectively detects communities in sparse networks, outperforming traditional algorithms and reaching theoretical detection limits.

Contribution

The paper presents a novel spectral algorithm using non-backtracking operators that improves community detection in sparse networks and achieves optimal performance.

Findings

The non-backtracking spectrum remains well-separated in sparse graphs.

The algorithm detects communities down to the theoretical limit.

Real-world network spectra show advantages over traditional methods.

Abstract

Spectral algorithms are classic approaches to clustering and community detection in networks. However, for sparse networks the standard versions of these algorithms are suboptimal, in some cases completely failing to detect communities even when other algorithms such as belief propagation can do so. Here we introduce a new class of spectral algorithms based on a non-backtracking walk on the directed edges of the graph. The spectrum of this operator is much better-behaved than that of the adjacency matrix or other commonly used matrices, maintaining a strong separation between the bulk eigenvalues and the eigenvalues relevant to community structure even in the sparse case. We show that our algorithm is optimal for graphs generated by the stochastic block model, detecting communities all the way down to the theoretical limit. We also show the spectrum of the non-backtracking operator for some real-world networks, illustrating its advantages over traditional spectral clustering.