Leaders, Followers, and Community Detection

TL;DR

This paper introduces a new community detection algorithm, LFA, based on a generative model, which outperforms existing methods in accuracy and scalability, especially on social network graphs like IMDB.

Contribution

The paper presents a new generative model for community formation and a leader-follower algorithm that guarantees exact recovery on this model, improving accuracy and efficiency.

Findings

LFA achieves an F1 score of 0.81 on IMDB graph.

LFA outperforms popular algorithms in accuracy.

FLFA is highly scalable with near-linear complexity.

Abstract

Communities in social networks or graphs are sets of well-connected, overlapping vertices. The effectiveness of a community detection algorithm is determined by accuracy in finding the ground-truth communities and ability to scale with the size of the data. In this work, we provide three contributions. First, we show that a popular measure of accuracy known as the F1 score, which is between 0 and 1, with 1 being perfect detection, has an information lower bound is 0.5. We provide a trivial algorithm that produces communities with an F1 score of 0.5 for any graph! Somewhat surprisingly, we find that popular algorithms such as modularity optimization, BigClam and CESNA have F1 scores less than 0.5 for the popular IMDB graph. To rectify this, as the second contribution we propose a generative model for community formation, the sequential community graph, which is motivated by the formation of social networks. Third, motivated by our generative model, we propose the leader-follower algorithm (LFA). We prove that it recovers all communities for sequential community graphs by establishing a structural result that sequential community graphs are chordal. For a large number of popular social networks, it recovers communities with a much higher F1 score than other popular algorithms. For the IMDB graph, it obtains an F1 score of 0.81. We also propose a modification to the LFA called the fast leader-follower algorithm (FLFA) which in addition to being highly accurate, is also fast, with a scaling that is almost linear in the network size.