Coded Sparse Matrix Multiplication

TL;DR

This paper introduces a new sparse coding strategy for distributed matrix multiplication that preserves sparsity, reduces decoding time, and improves efficiency in large-scale machine learning tasks.

Contribution

The paper proposes a novel sparse coding scheme that maintains matrix sparsity, achieves near-optimal recovery thresholds, and ensures linear decoding time, outperforming existing methods.

Findings

Achieves near optimal recovery threshold

Decoding time is linear in the number of non-zero elements

Demonstrates advantages over existing coded strategies

Abstract

In a large-scale and distributed matrix multiplication problem $C=A^{\intercal}B$, where $C\in\mathbb{R}^{r\times t}$, the coded computation plays an important role to effectively deal with "stragglers" (distributed computations that may get delayed due to few slow or faulty processors). However, existing coded schemes could destroy the significant sparsity that exists in large-scale machine learning problems, and could result in much higher computation overhead, i.e., $O(rt)$ decoding time. In this paper, we develop a new coded computation strategy, we call \emph{sparse code}, which achieves near \emph{optimal recovery threshold}, \emph{low computation overhead}, and \emph{linear decoding time} $O(nnz(C))$. We implement our scheme and demonstrate the advantage of the approach over both uncoded and current fastest coded strategies.