Determining the Number of Communities in Sparse and Imbalanced Settings

TL;DR

This paper introduces a spectral method using a novel network operator to accurately determine the number of communities in sparse and imbalanced networks, overcoming limitations of existing approaches.

Contribution

The authors develop a centered non-backtracking spectral operator that improves community detection in challenging network conditions, with theoretical and numerical validation.

Findings

Effective in ultra-sparse networks

Handles community size and density imbalance

Provides a reliable goodness-of-fit test

Abstract

Community structures represent a crucial aspect of network analysis, and various methods have been developed to identify these communities. However, a common hurdle lies in determining the number of communities K, a parameter that often requires estimation in practice. Existing approaches for estimating K face two notable challenges: the weak community signal present in sparse networks and the imbalance in community sizes or edge densities that result in unequal per-community expected degree. We propose a spectral method based on a novel network operator whose spectral properties effectively overcome both challenges. This operator is a refined version of the non-backtracking operator, adapted from a "centered" adjacency matrix. Its leading eigenvalues are more concentrated than those of the adjacency matrix for sparse networks, while they also demonstrate enhanced signal under imbalance scenarios, a benefit attributed to the centering step. This is justified, either theoretically or numerically, under the null model K = 1, in both dense and ultra-sparse settings. A goodness-of-fit test based on the leading eigenvalue can be applied to determine the number of communities K.