External Memory based Distributed Generation of Massive Scale Social Networks on Small Clusters

TL;DR

This paper presents a novel distributed external memory algorithm for generating massive-scale social network graphs on small clusters with limited main memory, significantly reducing resource requirements.

Contribution

It introduces new distributed external memory algorithms for graph generation, enabling large graphs to be created with minimal main memory and fewer compute nodes.

Findings

Able to generate a 2^38 node graph with only 64 nodes

Reduces resource needs compared to existing schemes

Applicable to SSD-based supercomputers and external memory graph libraries

Abstract

Small distributed systems are limited by their main memory to generate massively large graphs. Trivial extension to current graph generators to utilize external memory leads to large amount of random I/O hence do not scale with size. In this work we offer a technique to generate massive scale graphs on small cluster of compute nodes with limited main memory. We develop several distributed and external memory algorithms, primarily, shuffle, relabel, redistribute, and, compressed-sparse-row (csr) convert. The algorithms are implemented in MPI/pthread model to help parallelize the operations across multicores within each core. Using our scheme it is feasible to generate a graph of size $2^{38}$ nodes (scale 38) using only 64 compute nodes. This can be compared with the current scheme would require at least 8192 compute node, assuming 64GB of main memory. Our work has broader implications for external memory graph libraries such as STXXL and graph processing on SSD-based supercomputers such as Dash and Gordon [1][2].