Low-Bandwidth Matrix Multiplication: Faster Algorithms and More General Forms of Sparsity

TL;DR

This paper improves distributed algorithms for sparse matrix multiplication, reducing communication rounds, and extends the analysis to more general sparsity structures beyond uniform sparsity assumptions.

Contribution

It presents faster algorithms for uniformly sparse matrices and explores algorithms for various non-uniform sparsity models in distributed settings.

Findings

Improved round complexity from $O(d^{1.907})$ to $O(d^{1.832})$ for uniformly sparse matrices.

Extended analysis to row-sparse, column-sparse, and matrices with bounded degeneracy.

Provided insights into algorithms for general and average-sparse matrices.

Abstract

In prior work, Gupta et al. (SPAA 2022) presented a distributed algorithm for multiplying sparse $n \times n$ matrices, using $n$ computers. They assumed that the input matrices are uniformly sparse--there are at most $d$ non-zeros in each row and column--and the task is to compute a uniformly sparse part of the product matrix. The sparsity structure is globally known in advance (this is the supported setting). As input, each computer receives one row of each input matrix, and each computer needs to output one row of the product matrix. In each communication round each computer can send and receive one $O(\log n)$-bit message. Their algorithm solves this task in $O(d^{1.907})$ rounds, while the trivial bound is $O(d^2)$. We improve on the prior work in two dimensions: First, we show that we can solve the same task faster, in only $O(d^{1.832})$ rounds. Second, we explore what happens when matrices are not uniformly sparse. We consider the following alternative notions of sparsity: row-sparse matrices (at most $d$ non-zeros per row), column-sparse matrices, matrices with bounded degeneracy (we can recursively delete a row or column with at most $d$ non-zeros), average-sparse matrices (at most $dn$ non-zeros in total), and general matrices.