Communication-free Massively Distributed Graph Generation

TL;DR

This paper introduces scalable, communication-free graph generators that can produce massive synthetic networks efficiently, enabling large-scale analysis and benchmarking of complex networks.

Contribution

The authors develop novel, embarrassingly parallel graph generators using pseudorandomization and divide-and-conquer techniques, capable of generating extremely large networks.

Findings

Generated graphs with up to 2^43 vertices and 2^47 edges.

Achieved generation in less than 22 minutes on 32768 cores.

Demonstrated near-optimal scalability and realism of the generators.

Abstract

Analyzing massive complex networks yields promising insights about our everyday lives. Building scalable algorithms to do so is a challenging task that requires a careful analysis and an extensive evaluation. However, engineering such algorithms is often hindered by the scarcity of publicly~available~datasets. Network generators serve as a tool to alleviate this problem by providing synthetic instances with controllable parameters. However, many network generators fail to provide instances on a massive scale due to their sequential nature or resource constraints. Additionally, truly scalable network generators are few and often limited in their realism. In this work, we present novel generators for a variety of network models that are frequently used as benchmarks. By making use of pseudorandomization and divide-and-conquer schemes, our generators follow a communication-free paradigm. The resulting generators are thus embarrassingly parallel and have a near optimal scaling behavior. This allows us to generate instances of up to $2^{43}$ vertices and $2^{47}$ edges in less than 22 minutes on 32768 cores. Therefore, our generators allow new graph families to be used on an unprecedented scale.