Benchmarking community detection methods on social media data

TL;DR

This paper presents a new benchmark for community detection algorithms on social media data, highlighting the gap between small curated datasets and large real-world networks, and demonstrating the effectiveness of a task-based evaluation scheme.

Contribution

It introduces a task-based benchmarking strategy for large social media networks and applies it to Facebook data, revealing limitations of popular algorithms.

Findings

Popular algorithms struggle with fine-grained community detection

Benchmarking on Facebook data shows performance gaps

Evaluation scheme addresses challenges of large network data

Abstract

Benchmarking the performance of community detection methods on empirical social network data has been identified as critical for improving these methods. In particular, while most current research focuses on detecting communities in data that has been digitally extracted from large social media and telecommunications services, most evaluation of this research is based on small, hand-curated datasets. We argue that these two types of networks differ so significantly that by evaluating algorithms solely on the former, we know little about how well they perform on the latter. To address this problem, we consider the difficulties that arise in constructing benchmarks based on digitally extracted network data, and propose a task-based strategy which we feel addresses these difficulties. To demonstrate that our scheme is effective, we use it to carry out a substantial benchmark based on Facebook data. The benchmark reveals that some of the most popular algorithms fail to detect fine-grained community structure.