Partitioning Trillion Edge Graphs on Edge Devices

TL;DR

This paper presents StreamCPI, a new framework that enables high-quality partitioning of trillion-edge graphs on edge devices by reducing memory overhead through run-length compression, thus facilitating large-scale graph processing on low-cost hardware.

Contribution

StreamCPI introduces run-length compression of block assignments to significantly lower memory usage in streaming graph partitioning, enabling trillion-edge graph partitioning on edge devices.

Findings

StreamCPI reduces memory usage compared to traditional methods.

It enables partitioning of graphs with over a trillion edges on low-cost devices.

StreamCPI achieves better partition quality than Hashing on large graphs.

Abstract

Processing large-scale graphs, containing billions of entities, is critical across fields like bioinformatics, high-performance computing, navigation and route planning, among others. Efficient graph partitioning, which divides a graph into sub-graphs while minimizing inter-block edges, is essential to graph processing, as it optimizes parallel computing and enhances data locality. Traditional in-memory partitioners, such as METIS and KaHIP, offer high-quality partitions but are often infeasible for enormous graphs due to their substantial memory overhead. Streaming partitioners reduce memory usage to O(n), where 'n' is the number of nodes of the graph, by loading nodes sequentially and assigning them to blocks on-the-fly. This paper introduces StreamCPI, a novel framework that further reduces the memory overhead of streaming partitioners through run-length compression of block assignments. Notably, StreamCPI enables the partitioning of trillion-edge graphs on edge devices. Additionally, within this framework, we propose a modification to the LA-vector bit vector for append support, which can be used for online run-length compression in other streaming applications. Empirical results show that StreamCPI reduces memory usage while maintaining or improving partition quality. For instance, using StreamCPI, the Fennel partitioner effectively partitions a graph with 17 billion nodes and 1.03 trillion edges on a Raspberry Pi, achieving significantly better solution quality than Hashing, the only other feasible algorithm on edge devices. StreamCPI thus advances graph processing by enabling high-quality partitioning on low-cost machines.