Efficient Distributed Transposition Of Large-Scale Multigraphs And High-Cardinality Sparse Matrices

TL;DR

This paper introduces a novel distributed transposition algorithm for large-scale multigraphs and high-cardinality sparse matrices, addressing a gap in existing methods and demonstrating excellent scalability.

Contribution

It provides a theoretical model, algorithmic implementation, and proof of correctness for transposing complex multigraph adjacency matrices in distributed environments.

Findings

Achieves near-ideal scaling on balanced datasets

Demonstrates effective handling of heterogeneous data distributions

Advances state-of-the-art in distributed matrix transposition

Abstract

Graph-based representations underlie a wide range of scientific problems. Graph connectivity is typically represented as a sparse matrix in the Compressed Sparse Row format. Large-scale graphs rely on distributed storage, allocating distinct subsets of rows to compute nodes. Efficient matrix transpose is an operation of high importance, providing the reverse graph pathways and a column-ordered matrix view. This operation is well studied for simple graph models. Nevertheless, its resolution for multigraphs and higher-cardinality connectivity matrices is unexistent. We advance state-of-the-art distributed transposition methods by providing a theoretical model, algorithmic details, MPI-based implementation and proof of mathematical soundness for such complex models. Benchmark results demonstrate ideal and almost ideal scaling properties for perfectly- and heterogeneously-balanced datasets, respectively