Characterizing and comparing external measures for the assessment of cluster analysis and community detection

TL;DR

This paper introduces an empirical framework to evaluate and compare external measures for assessing cluster analysis and community detection, aiding users in selecting appropriate measures based on their behavior under various transformations.

Contribution

It proposes a novel, application-independent evaluation method that describes and compares external measures through regression analysis of their responses to parametric transformations.

Findings

Framework effectively characterizes measure behavior

Enables comparison of different evaluation measures

Assists in selecting suitable measures for specific applications

Abstract

In the context of cluster analysis and graph partitioning, many external evaluation measures have been proposed in the literature to compare two partitions of the same set. This makes the task of selecting the most appropriate measure for a given situation a challenge for the end user. However, this issue is overlooked in the literature. Researchers tend to follow tradition and use the standard measures of their field, although they often became standard only because previous researchers started consistently using them. In this work, we propose a new empirical evaluation framework to solve this issue, and help the end user selecting an appropriate measure for their application. For a collection of candidate measures, it first consists in describing their behavior by computing them for a generated dataset of partitions, obtained by applying a set of predefined parametric partition transformations. Second, our framework performs a regression analysis to characterize the measures in terms of how they are affected by these parameters and transformations. This allows both describing and comparing the measures. Our approach is not tied to any specific measure or application, so it can be applied to any situation. We illustrate its relevance by applying it to a selection of standard measures, and show how it can be put in practice through two concrete use cases.