Large-Scale Graphs Community Detection using Spark GraphFrames

TL;DR

This paper presents a scalable framework for large-scale community detection in graphs using Apache Spark GraphFrames, demonstrating its effectiveness on real-world datasets.

Contribution

Introduces a novel Spark GraphFrames-based framework for community detection algorithms like K-Cliques, Louvain, and Fast Greedy, addressing scalability issues.

Findings

Framework is feasible for large-scale graphs

Algorithms perform well on real-world datasets

Scalability improves with distributed processing

Abstract

With the emergence of social networks, online platforms dedicated to different use cases, and sensor networks, the emergence of large-scale graph community detection has become a steady field of research with real-world applications. Community detection algorithms have numerous practical applications, particularly due to their scalability with data size. Nonetheless, a notable drawback of community detection algorithms is their computational intensity~\cite{Apostol2014}, resulting in decreasing performance as data size increases. For this purpose, new frameworks that employ distributed systems such as Apache Hadoop and Apache Spark which can seamlessly handle large-scale graphs must be developed. In this paper, we propose a novel framework for community detection algorithms, i.e., K-Cliques, Louvain, and Fast Greedy, developed using Apache Spark GraphFrames. We test their performance and scalability on two real-world datasets. The experimental results prove the feasibility of developing graph mining algorithms using Apache Spark GraphFrames.