Sparsity-Preserving Encodings for Straggler-Optimal Distributed Matrix Computations at the Edge

TL;DR

This paper develops sparsity-preserving coding schemes for distributed matrix computations at the edge, achieving straggler resilience while maintaining the sparsity of matrices to enhance computational efficiency.

Contribution

It introduces a lower bound on coding weight for straggler resilience and proposes schemes that meet this bound while preserving matrix sparsity.

Findings

Numerical experiments on AWS validate straggler mitigation.

Proposed schemes retain matrix sparsity and improve computation speed.

Achieves maximum straggler resilience with minimal coding overhead.

Abstract

Matrix computations are a fundamental building-block of edge computing systems, with a major recent uptick in demand due to their use in AI/ML training and inference procedures. Existing approaches for distributing matrix computations involve allocating coded combinations of submatrices to worker nodes, to build resilience to slower nodes, called stragglers. In the edge learning context, however, these approaches will compromise sparsity properties that are often present in the original matrices found at the edge server. In this study, we consider the challenge of augmenting such approaches to preserve input sparsity when distributing the task across edge devices, thereby retaining the associated computational efficiency enhancements. First, we find a lower bound on the weight of coding, i.e., the number of submatrices to be combined to obtain coded submatrices, to provide the resilience to the maximum possible number of straggler devices (for given number of devices and their storage constraints). Next we propose distributed matrix computation schemes which meet the exact lower bound on the weight of the coding. Numerical experiments conducted in Amazon Web Services (AWS) validate our assertions regarding straggler mitigation and computation speed for sparse matrices.