Recursive Multi-Section on the Fly: Shared-Memory Streaming Algorithms for Hierarchical Graph Partitioning and Process Mapping

TL;DR

This paper introduces a shared-memory streaming recursive multi-section algorithm for hierarchical graph partitioning and process mapping, achieving faster performance and better mappings than existing methods, suitable for large-scale distributed systems.

Contribution

It presents a novel on-the-fly recursive multi-section streaming algorithm that improves speed and mapping quality for hierarchical graph partitioning.

Findings

Up to two orders of magnitude faster than Fennel.

Produces better process mappings than competing tools.

Achieves 5% more cut edges with significantly reduced runtime.

Abstract

Partitioning a graph into balanced blocks such that few edges run between blocks is a key problem for large-scale distributed processing. A current trend for partitioning huge graphs are streaming algorithms, which use low computational resources. In this work, we present a shared-memory streaming multi-recursive partitioning scheme that performs recursive multi-sections on the fly without knowing the overall input graph. Our approach has a considerably lower running time complexity in comparison with state-of-the-art non-buffered one-pass partitioning algorithms for the standard graph partitioning case. Moreover, if the topology of a distributed system is known, it is possible to further optimize the communication costs by mapping partitions onto processing elements. Our experiments indicate that our algorithm is both faster and produces better process mappings than competing tools. In case of graph partitioning, our framework is up to two orders of magnitude faster at the cost of 5% more cut edges compared to Fennel.