An Initial Evaluation of Distributed Graph Algorithms using NWGraph and HPX

TL;DR

This paper presents a distributed graph processing approach using NWGraph and HPX, aiming to reduce synchronization costs and improve scalability for large-scale graph algorithms like BFS and PageRank.

Contribution

It introduces a novel integration of NWGraph with the HPX runtime to leverage asynchronous tasks for distributed graph analytics, addressing synchronization and load balancing issues.

Findings

BFS outperforms distributed BGL in initial tests

PageRank currently does not outperform BGL

Highlights potential and challenges of asynchronous runtimes for graph processing

Abstract

Graphs are central to modeling relationships in scientific computing, data analysis, and AI/ML, but their growing scale can exceed the memory and compute capacity of single nodes, requiring distributed solutions. Existing distributed graph framework, however, face fundamental challenges: graph algorithms are latency-bound, suffer from irregular memory access, and often impose synchronization costs that limit scalability and efficiency. In this work, we present a distributed implementation of the NWGraph library integrated with the HPX runtime system. By leveraging HPX's asynchronous many-task model, our approach aims to reduce synchronization overhead, improve load balance, and provide a foundation for distributed graph analytics. We evaluate this approach using two representative algorithms: Breadth-First-Search (BFS) and (PageRank). Our initial results show that BFS achieves better performance than the distributed Boost Graph Library (BGL), while PageRank remains more challenging, with current implementation not yet outperforming BGL. These findings highlight both the promise and the open challenges of applying asynchronous task-based runtimes to graph processing, and point to opportunities for future optimizations and extensions.