An Adaptive Parallel Algorithm for Computing Connected Components

TL;DR

This paper introduces a hybrid parallel algorithm for connected components that adapts to graph topology, significantly improving efficiency and scalability on large, diverse graphs using distributed memory systems.

Contribution

It proposes a novel hybrid approach combining parallel BFS and other techniques, with a heuristic to select the optimal method based on graph topology.

Findings

Achieves 24x speedup over previous methods.

Efficiently processes graphs with over 50 billion edges.

Scalable performance on 32K cores of Cray XC30.

Abstract

We present an efficient distributed memory parallel algorithm for computing connected components in undirected graphs based on Shiloach-Vishkin's PRAM approach. We discuss multiple optimization techniques that reduce communication volume as well as load-balance the algorithm. We also note that the efficiency of the parallel graph connectivity algorithm depends on the underlying graph topology. Particularly for short diameter graph components, we observe that parallel Breadth First Search (BFS) method offers better performance. However, running parallel BFS is not efficient for computing large diameter components or large number of small components. To address this challenge, we employ a heuristic that allows the algorithm to quickly predict the type of the network by computing the degree distribution and follow the optimal hybrid route. Using large graphs with diverse topologies from domains including metagenomics, web crawl, social graph and road networks, we show that our hybrid implementation is efficient and scalable for each of the graph types. Our approach achieves a runtime of 215 seconds using 32K cores of Cray XC30 for a metagenomic graph with over 50 billion edges. When compared against the previous state-of-the-art method, we see performance improvements up to 24x.