Evaluation of Community Detection Methods

TL;DR

This paper critically evaluates existing metrics for community detection quality, identifies their limitations, and proposes a new evaluation method based on integer linear programming, kappa index, and F-score, demonstrating improved assessment accuracy.

Contribution

The paper introduces a novel evaluation approach for community detection that overcomes limitations of NMI-based metrics by using integer linear programming, kappa index, and F-score.

Findings

cNMI violates proportionality assumption

NMI-type metrics ignore small communities

Proposed method shows improved evaluation accuracy

Abstract

Community structures are critical towards understanding not only the network topology but also how the network functions. However, how to evaluate the quality of detected community structures is still challenging and remains unsolved. The most widely used metric, normalized mutual information (NMI), was proved to have finite size effect, and its improved form relative normalized mutual information (rNMI) has reverse finite size effect. Corrected normalized mutual information (cNMI) was thus proposed and has neither finite size effect nor reverse finite size effect. However, in this paper we show that cNMI violates the so-called proportionality assumption. In addition, NMI-type metrics have the problem of ignoring importance of small communities. Finally, they cannot be used to evaluate a single community of interest. In this paper, we map the computed community labels to the ground-truth ones through integer linear programming, then use kappa index and F-score to evaluate the detected community structures. Experimental results demonstrate the advantages of our method.