On Optimizing Resource Utilization in Distributed Connected Components

TL;DR

This paper introduces two new distributed algorithms, SiskinCC and RobinCC, that significantly improve resource utilization and speed up connected components computation on large-scale graphs by optimizing memory and network bandwidth.

Contribution

The paper presents novel distributed CC algorithms that optimize memory and network bandwidth, enabling faster processing of massive graphs compared to existing methods.

Findings

SiskinCC and RobinCC achieve 29.1x and 16.8x speedups respectively.

Algorithms efficiently utilize shared memory and graph structure.

Effective on graphs with up to 500 billion edges and 11.7 billion vertices.

Abstract

Connected Components (CC) is a core graph problem with numerous applications. This paper investigates accelerating distributed CC by optimizing memory and network bandwidth utilization. We present two novel distributed CC algorithms, SiskinCC and RobinCC, which are built upon the Jayanti-Tarjan disjoint set union algorithm. To optimize memory utilization, SiskinCC and RobinCC are designed to facilitate efficient access to a shared array for all cores running in a machine. This allows execution of faster algorithms with larger memory bounds. SiskinCC leverages the continuous inter-machine communication during the computation phase to reduce the final communication overhead and RobinCC leverages the structural properties of real-world graphs to optimize network bandwidth utilization. Our evaluation against a distributed state-of-the-art CC algorithm, using real-world and synthetic graphs with up to 500 billion edges and 11.7 billion vertices, and on up to 2048 CPU cores, demonstrates that SiskinCC and RobinCC achieve geometric mean speedups of 29.1 and 16.8 times.