Normalized Mutual Information to evaluate overlapping community finding algorithms

TL;DR

This paper evaluates the use of normalized mutual information for measuring the accuracy of overlapping community detection algorithms, highlighting issues with current normalization methods and proposing improvements.

Contribution

It identifies problems with existing normalized mutual information measures and proposes a corrected normalization approach for better accuracy in evaluating overlapping clustering.

Findings

Normalized mutual information can behave unintuitively with current normalization.

A more conventional normalization improves the measure's behavior.

Comparison with Omega index shows the effectiveness of the proposed normalization.

Abstract

Given the increasing popularity of algorithms for overlapping clustering, in particular in social network analysis, quantitative measures are needed to measure the accuracy of a method. Given a set of true clusters, and the set of clusters found by an algorithm, these sets of clusters must be compared to see how similar or different the sets are. A normalized measure is desirable in many contexts, for example assigning a value of 0 where the two sets are totally dissimilar, and 1 where they are identical. A measure based on normalized mutual information, [1], has recently become popular. We demonstrate unintuitive behaviour of this measure, and show how this can be corrected by using a more conventional normalization. We compare the results to that of other measures, such as the Omega index [2].