Finding community structure in very large networks

TL;DR

This paper introduces a fast hierarchical algorithm for detecting community structures in large, sparse networks, capable of analyzing networks with millions of edges efficiently, and demonstrates its effectiveness on a large online retailer network.

Contribution

The paper presents a novel hierarchical agglomeration algorithm with near-linear runtime for community detection in large networks, outperforming existing methods in speed and scalability.

Findings

Algorithm efficiently analyzes large networks with millions of edges.

Successfully identified meaningful communities in a large online retailer network.

Revealed large-scale patterns in customer purchasing habits.

Abstract

The discovery and analysis of community structure in networks is a topic of considerable recent interest within the physics community, but most methods proposed so far are unsuitable for very large networks because of their computational cost. Here we present a hierarchical agglomeration algorithm for detecting community structure which is faster than many competing algorithms: its running time on a network with n vertices and m edges is O(m d log n) where d is the depth of the dendrogram describing the community structure. Many real-world networks are sparse and hierarchical, with m ~ n and d ~ log n, in which case our algorithm runs in essentially linear time, O(n log^2 n). As an example of the application of this algorithm we use it to analyze a network of items for sale on the web-site of a large online retailer, items in the network being linked if they are frequently purchased by the same buyer. The network has more than 400,000 vertices and 2 million edges. We show that our algorithm can extract meaningful communities from this network, revealing large-scale patterns present in the purchasing habits of customers.