Scaling Betweenness Approximation to Billions of Edges by MPI-based Adaptive Sampling

TL;DR

This paper introduces an MPI-based adaptive sampling algorithm that significantly accelerates betweenness centrality approximation, enabling analysis of billion-edge graphs in under ten minutes with high accuracy.

Contribution

It presents the first MPI-parallelized adaptive sampling algorithm for betweenness approximation, achieving substantial speedups over shared-memory methods on large-scale graphs.

Findings

16.1x faster than shared-memory implementation for the sampling phase

7.4x average speedup for the complete algorithm

Able to approximate betweenness on graphs with billions of edges in under ten minutes

Abstract

Betweenness centrality is one of the most popular vertex centrality measures in network analysis. Hence, many (sequential and parallel) algorithms to compute or approximate betweenness have been devised. Recent algorithmic advances have made it possible to approximate betweenness very efficiently on shared-memory architectures. Yet, the best shared-memory algorithms can still take hours of running time for large graphs, especially for graphs with a high diameter or when a small relative error is required. In this work, we present an MPI-based generalization of the state-of-the-art shared-memory algorithm for betweenness approximation. This algorithm is based on adaptive sampling; our parallelization strategy can be applied in the same manner to adaptive sampling algorithms for other problems. In experiments on a 16-node cluster, our MPI-based implementation is by a factor of 16.1x faster than the state-of-the-art shared-memory implementation when considering our parallelization focus -- the adaptive sampling phase -- only. For the complete algorithm, we obtain an average (geom. mean) speedup factor of 7.4x over the state of the art. For some previously very challenging inputs, this speedup is much higher. As a result, our algorithm is the first to approximate betweenness centrality on graphs with several billion edges in less than ten minutes with high accuracy.