Distributed Estimation of Graph 4-Profiles

TL;DR

This paper introduces a distributed, message-passing algorithm for efficiently estimating global and local 4-profiles in large graphs, enabling rapid analysis of connectivity patterns with theoretical guarantees.

Contribution

The paper presents a novel distributed algorithm for counting 4-node subgraphs and estimating local 4-profiles using compressed two-hop information and graph sparsification techniques.

Findings

Algorithm computes global and local 4-profiles in minutes on graphs with millions of edges.

Scales up to 640 cores with significant speedup over previous methods.

Provides theoretical guarantees on approximation quality through concentration results.

Abstract

We present a novel distributed algorithm for counting all four-node induced subgraphs in a big graph. These counts, called the $4$-profile, describe a graph's connectivity properties and have found several uses ranging from bioinformatics to spam detection. We also study the more complicated problem of estimating the local $4$-profiles centered at each vertex of the graph. The local $4$-profile embeds every vertex in an $11$-dimensional space that characterizes the local geometry of its neighborhood: vertices that connect different clusters will have different local $4$-profiles compared to those that are only part of one dense cluster. Our algorithm is a local, distributed message-passing scheme on the graph and computes all the local $4$-profiles in parallel. We rely on two novel theoretical contributions: we show that local $4$-profiles can be calculated using compressed two-hop information and also establish novel concentration results that show that graphs can be substantially sparsified and still retain good approximation quality for the global $4$-profile. We empirically evaluate our algorithm using a distributed GraphLab implementation that we scaled up to $640$ cores. We show that our algorithm can compute global and local $4$-profiles of graphs with millions of edges in a few minutes, significantly improving upon the previous state of the art.