On Parallelizing Matrix Multiplication by the Column-Row Method

TL;DR

This paper introduces a simple, communication-efficient parallel method for matrix multiplication that guarantees consistent output distribution, enabling scalable processing of large matrices and applications like frequent item mining.

Contribution

The authors present a novel consistent parallelization technique for matrix multiplication that minimizes communication and balances workload across processors.

Findings

Achieves linear speedup until input reading dominates cost.

Provides theoretical guarantees on per-processor work.

Demonstrates effectiveness on real-world data for frequent pair mining.

Abstract

We consider the problem of sparse matrix multiplication by the column row method in a distributed setting where the matrix product is not necessarily sparse. We present a surprisingly simple method for "consistent" parallel processing of sparse outer products (column-row vector products) over several processors, in a communication-avoiding setting where each processor has a copy of the input. The method is consistent in the sense that a given output entry is always assigned to the same processor independently of the specific structure of the outer product. We show guarantees on the work done by each processor, and achieve linear speedup down to the point where the cost is dominated by reading the input. Our method gives a way of distributing (or parallelizing) matrix product computations in settings where the main bottlenecks are storing the result matrix, and inter-processor communication. Motivated by observations on real data that often the absolute values of the entries in the product adhere to a power law, we combine our approach with frequent items mining algorithms and show how to obtain a tight approximation of the weight of the heaviest entries in the product matrix. As a case study we present the application of our approach to frequent pair mining in transactional data streams, a problem that can be phrased in terms of sparse ${0,1}$-integer matrix multiplication by the column-row method. Experimental evaluation of the proposed method on real-life data supports the theoretical findings.