ACGraph: An Efficient Asynchronous Out-of-Core Graph Processing Framework

TL;DR

ACGraph is a new asynchronous out-of-core graph processing framework optimized for SSDs, which reduces I/O overhead and synchronization delays, enabling faster execution of large-scale graph algorithms beyond main memory capacity.

Contribution

It introduces a dynamic priority scheduler and an online asynchronous worklist, unifying I/O and computation to improve efficiency over existing synchronous systems.

Findings

Significantly faster runtime compared to state-of-the-art systems.

Reduced I/O operations through efficient block reuse.

Effective processing of large graphs with limited memory.

Abstract

Graphs are a ubiquitous data structure in diverse domains such as machine learning, social networks, and data mining. As real-world graphs continue to grow beyond the memory capacity of single machines, out-of-core graph processing systems have emerged as a viable solution. Yet, existing systems that rely on strictly synchronous, iteration-by-iteration execution incur significant overheads. In particular, their scheduling mechanisms lead to I/O inefficiencies, stemming from read and work amplification, and induce costly synchronization stalls hindering sustained disk utilization. To overcome these limitations, we present {\em ACGraph}, a novel asynchronous graph processing system optimized for SSD-based environments with constrained memory resources. ACGraph employs a dynamic, block-centric priority scheduler that adjusts in real time based on workload, along with an online asynchronous worklist that minimizes redundant disk accesses by efficiently reusing active blocks in memory. Moreover, ACGraph unifies asynchronous I/O with computation in a pipelined execution model that maintains sustained I/O activation, and leverages a highly optimized hybrid storage format to expedite access to low-degree vertices. We implement popular graph algorithms, such as Breadth-First Search (BFS), Weakly Connected Components (WCC), personalized PageRank (PPR), PageRank (PR), and $k$-core on ACGraph and demonstrate that ACGraph substantially outperforms state-of-the-art out-of-core graph processing systems in both runtime and I/O efficiency.