Sparse Random Khatri-Rao Product Codes for Distributed Matrix Multiplication

TL;DR

This paper proposes Sparse Random Khatri-Rao Product (SRKRP) codes for distributed matrix multiplication, reducing costs for sparse matrices while maintaining stability, and enhances failure probability with minimal additional computations.

Contribution

Introduction of SRKRP codes with sparse generator matrices and analysis of their properties, along with a method to improve failure probability via extra small computations.

Findings

SRKRP codes reduce encoding and communication costs for sparse matrices.

The probability of rank deficiency depends on sparsity and non-straggler count.

Additional small computations significantly lower failure probability.

Abstract

We introduce two generalizations to the paradigm of using Random Khatri-Rao Product (RKRP) codes for distributed matrix multiplication. We first introduce a class of codes called Sparse Random Khatri-Rao Product (SRKRP) codes which have sparse generator matrices. SRKRP codes result in lower encoding, computation and communication costs than RKRP codes when the input matrices are sparse, while they exhibit similar numerical stability to other state of the art schemes. We empirically study the relationship between the probability of the generator matrix (restricted to the set of non-stragglers) of a randomly chosen SRKRP code being rank deficient and various parameters of the coding scheme including the degree of sparsity of the generator matrix and the number of non-stragglers. Secondly, we show that if the master node can perform a very small number of matrix product computations in addition to the computations performed by the workers, the failure probability can be substantially improved.