Partitioning Trillion Edge Graphs on Edge Devices
Adil Chhabra, Florian Kurpicz, Christian Schulz, Dominik Schweisgut,, Daniel Seemaier

TL;DR
This paper presents StreamCPI, a new framework that enables high-quality partitioning of trillion-edge graphs on edge devices by reducing memory overhead through run-length compression, thus facilitating large-scale graph processing on low-cost hardware.
Contribution
StreamCPI introduces run-length compression of block assignments to significantly lower memory usage in streaming graph partitioning, enabling trillion-edge graph partitioning on edge devices.
Findings
StreamCPI reduces memory usage compared to traditional methods.
It enables partitioning of graphs with over a trillion edges on low-cost devices.
StreamCPI achieves better partition quality than Hashing on large graphs.
Abstract
Processing large-scale graphs, containing billions of entities, is critical across fields like bioinformatics, high-performance computing, navigation and route planning, among others. Efficient graph partitioning, which divides a graph into sub-graphs while minimizing inter-block edges, is essential to graph processing, as it optimizes parallel computing and enhances data locality. Traditional in-memory partitioners, such as METIS and KaHIP, offer high-quality partitions but are often infeasible for enormous graphs due to their substantial memory overhead. Streaming partitioners reduce memory usage to O(n), where 'n' is the number of nodes of the graph, by loading nodes sequentially and assigning them to blocks on-the-fly. This paper introduces StreamCPI, a novel framework that further reduces the memory overhead of streaming partitioners through run-length compression of block…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGraph Labeling and Dimension Problems · Graph Theory and Algorithms · Optimization and Search Problems
