StarDist: A Code Generator for Distributed Graph Algorithms

TL;DR

StarDist introduces a distributed graph algorithm framework that optimizes communication and memory access, significantly improving performance on large-scale graph processing tasks.

Contribution

It presents a novel analysis-transformation framework and an optimized RMA-based reduction substrate for scalable distributed graph algorithms.

Findings

Outperforms d-Galois by 2.05x in SSSP

Outperforms DRONE by 1.44x in SSSP

Provides scalable distributed graph algorithm implementation

Abstract

Relational data, occurring in the real world, are often structured as graphs, which provide the logical abstraction required to make analytical derivations simpler. As graphs get larger, the irregular access patterns exhibited in most graph algorithms, hamper performance. This, along with NUMA and physical memory limits, results in scaling complexities with sequential/shared memory frameworks. StarPlat's MPI backend abstracts away the programmatic complexity involved in designing optimal distributed graph algorithms. It provides an instrument for coding graph algorithms that scale over distributed memory. In this work, we provide an analysis-transformation framework that leverages general semantics associated with iterations involving nodes and their neighbors, within StarPlat, to aggregate communication. The framework scans for patterns that warrant re-ordering in neighborhood access patterns, aggregate communication, and avoid communication altogether with opportunistic caching in reduction constructs. We also architect an optimized bulk-reduction substrate using Open MPI's passive Remote Memory Access (RMA) constructs. We applied our optimization logic to StarPlat's distributed backend and outperformed d-Galois by 2.05 and DRONE by 1.44 times in Single Source Shortest Paths across several big data graphs.