Accelerating Loading WebGraphs in ParaGrapher

TL;DR

This paper introduces two optimizations, PG-Fuse and CompBin, to enhance graph loading speed in ParaGrapher by improving storage utilization and decompression bandwidth, achieving significant speedups.

Contribution

The paper presents novel filesystem and data representation optimizations that significantly accelerate large-scale graph loading in ParaGrapher.

Findings

PG-Fuse achieves up to 7.6x speedup in graph loading.

CompBin achieves up to 21.8x speedup in decompression.

Combined, they enable efficient processing of graphs with up to 128 billion edges.

Abstract

ParaGrapher is a graph loading API and library that enables graph processing frameworks to load large-scale compressed graphs with minimal overhead. This capability accelerates the design and implementation of new high-performance graph algorithms and their evaluation on a wide range of graphs and across different frameworks. However, our previous study identified two major limitations in ParaGrapher: inefficient utilization of high-bandwidth storage and reduced decompression bandwidth due to increased compression ratios. To address these limitations, we present two optimizations for ParaGrapher in this paper. To improve storage utilization, particularly for high-bandwidth storage, we introduce ParaGrapher-FUSE (PG-Fuse) a filesystem based on the FUSE (Filesystem in User Space). PG-Fuse optimizes storage access by increasing the size of requested blocks, reducing the number of calls to the underlying filesystem, and caching the received blocks in memory for future calls. To improve the decompression bandwidth, we introduce CompBin, a compact binary representation of the CSR format. CompBin facilitates direct accesses to neighbors while preventing storage usage for unused bytes. Our evaluation on 12 real-world and synthetic graphs with up to 128 billion edges shows that PG-Fuse and CompBin achieve up to 7.6 and 21.8 times speedup, respectively.