Detecting communities using asymptotical Surprise

TL;DR

This paper introduces an asymptotic approximation of the surprise measure for community detection, enabling efficient optimization and comparison with modularity, especially in networks with many small communities.

Contribution

The authors develop an asymptotic approximation of surprise, facilitating efficient algorithms and extending its application to weighted graphs, while analyzing its advantages over modularity.

Findings

Surprise is less affected by the resolution limit than modularity.

Surprise tends to overestimate the number of communities.

Modularity performs better with fewer, larger communities.

Abstract

Nodes in real-world networks are repeatedly observed to form dense clusters, often referred to as communities. Methods to detect these groups of nodes usually maximize an objective function, which implicitly contains the definition of a community. We here analyze a recently proposed measure called surprise, which assesses the quality of the partition of a network into communities. In its current form, the formulation of surprise is rather difficult to analyze. We here therefore develop an accurate asymptotic approximation. This allows for the development of an efficient algorithm for optimizing surprise. Incidentally, this leads to a straightforward extension of surprise to weighted graphs. Additionally, the approximation makes it possible to analyze surprise more closely and compare it to other methods, especially modularity. We show that surprise is (nearly) unaffected by the well known resolution limit, a particular problem for modularity. However, surprise may tend to overestimate the number of communities, whereas they may be underestimated by modularity. In short, surprise works well in the limit of many small communities, whereas modularity works better in the limit of few large communities. In this sense, surprise is more discriminative than modularity, and may find communities where modularity fails to discern any structure.