Distributed Community Detection with the WCC Metric

TL;DR

This paper introduces a novel distributed, vertex-centric algorithm for community detection using the WCC metric, demonstrating high scalability and performance on large real-world graphs with up to 1.8 billion vertices.

Contribution

It presents the first distributed algorithm for optimizing the WCC metric, enhancing scalability and efficiency in large-scale community detection tasks.

Findings

Algorithm scales well with large graphs

Achieves superior results over traditional metrics

Operates efficiently on up to 32 machines

Abstract

Community detection has become an extremely active area of research in recent years, with researchers proposing various new metrics and algorithms to address the problem. Recently, the Weighted Community Clustering (WCC) metric was proposed as a novel way to judge the quality of a community partitioning based on the distribution of triangles in the graph, and was demonstrated to yield superior results over other commonly used metrics like modularity. The same authors later presented a parallel algorithm for optimizing WCC on large graphs. In this paper, we propose a new distributed, vertex-centric algorithm for community detection using the WCC metric. Results are presented that demonstrate the algorithm's performance and scalability on up to 32 worker machines and real graphs of up to 1.8 billion vertices. The algorithm scales best with the largest graphs, and to our knowledge, it is the first distributed algorithm for optimizing the WCC metric.