Random Alloy Codes and the Fundamental Limits of Coded Distributed Tensors

TL;DR

This paper introduces a probabilistic approach to coded distributed tensor computations, leading to the development of locally random alloy codes that optimize decoding probability and reveal fundamental limits of such coding schemes.

Contribution

It proposes a new probabilistic framework for coded tensor computations, resulting in locally random alloy codes and an impossibility theorem for coded distributed tensors.

Findings

Locally random alloy codes are optimal for decoding probability.

A fundamental impossibility theorem for coded distributed tensors.

Probabilistic approach improves over traditional combinatorial methods.

Abstract

Tensors are a fundamental operation in distributed computing, \emph{e.g.,} machine learning, that are commonly distributed into multiple parallel tasks for large datasets. Stragglers and other failures can severely impact the overall completion time. Recent works in coded computing provide a novel strategy to mitigate stragglers with coded tasks, with an objective of minimizing the number of tasks needed to recover the overall result, known as the recovery threshold. However, we demonstrate that this strict combinatorial definition does not directly optimize the probability of failure. In this paper, we focus on the most likely event and measure the optimality of a coding scheme more directly by its probability of decoding. Our probabilistic approach leads us to a practical construction of random codes for matrix multiplication, i.e., locally random alloy codes, which are optimal with respect to the measures. Furthermore, the probabilistic approach allows us to discover a surprising impossibility theorem about both random and deterministic coded distributed tensors.