Asymptotically Optimal Coded Distributed Computing via Combinatorial Designs

TL;DR

This paper introduces a new combinatorial design-based approach for coded distributed computing that reduces data splitting complexity and communication load, operating efficiently over binary fields and applicable to various parameters.

Contribution

It develops asymptotically optimal CDC schemes using combinatorial t-designs, significantly reducing data and function counts compared to prior methods.

Findings

Smaller input file and Reduce function numbers than existing schemes.

Achieves lower communication loads while maintaining optimality.

Operates over the binary field 2, unlike previous schemes.

Abstract

Coded distributed computing (CDC) introduced by Li \emph{et al.} can greatly reduce the communication load for MapReduce computing systems. In the general cascaded CDC with $K$ workers, $N$ input files and $Q$ Reduce functions, each input file will be mapped by $r$ workers and each Reduce function will be computed by $s$ workers such that coding techniques can be applied to achieve the maximum multicast gain. The main drawback of most existing CDC schemes is that they require the original data to be split into a large number of input files that grows exponentially with $K$, which can significantly increase the coding complexity and degrade system performance. In this paper, we first use a classic combinatorial structure $t$-design, for any integer $t\geq 2$, to develop a low-complexity and asymptotically optimal CDC with $r=s$. The main advantages of our scheme via $t$-design are two-fold: 1) having much smaller $N$ and $Q$ than the existing schemes under the same parameters $K$, $r$ and $s$; and 2) achieving smaller communication loads compared with the state-of-the-art schemes. Remarkably, unlike the previous schemes that realize on large operation fields, our scheme operates on the minimum binary field $\mathbb{F}_2$. Furthermore, we show that our construction method can incorporate the other combinatorial structures that have a similar property to $t$-design. For instance, we use $t$-GDD to obtain another asymptotically optimal CDC scheme over $\mathbb{F}_2$ that has different parameters from $t$-design. Finally, we show that our construction method can also be used to construct CDC schemes with $r\neq s$ that have small file number and Reduce function number.