Universal and Asymptotically Optimal Data and Task Allocation in Distributed Computing

TL;DR

This paper introduces a deterministic data and task allocation method for distributed computing that optimally balances communication and computation costs, scalable across various function decompositions without reshuffling files.

Contribution

The paper proposes the Interweaved-Cliques (IC) design, a novel deterministic allocation scheme that achieves order-optimal costs for a broad class of function decompositions in distributed computing.

Findings

Achieves order-optimal communication cost scaling as n/N^{1/d}

Provides a deterministic allocation method that is blind to specific function details

Enables multiple functions to be computed without reshuffling files

Abstract

We study the joint minimization of communication and computation costs in distributed computing, where a master node coordinates $N$ workers to evaluate a function over a library of $n$ files. Assuming that the function is decomposed into an arbitrary subfunction set $\mathbf{X}$, with each subfunction depending on $d$ input files, renders our distributed computing problem into a $d$-uniform hypergraph edge partitioning problem wherein the edge set (subfunction set), defined by $d$-wise dependencies between vertices (files) must be partitioned across $N$ disjoint groups (workers). The aim is to design a file and subfunction allocation, corresponding to a partition of $\mathbf{X}$, that minimizes the communication cost $\pi_{\mathbf{X}}$, representing the maximum number of distinct files per server, while also minimizing the computation cost $\delta_{\mathbf{X}}$ corresponding to a maximal worker subfunction load. For a broad range of parameters, we propose a deterministic allocation solution, the \emph{Interweaved-Cliques (IC) design}, whose information-theoretic-inspired interweaved clique structure simultaneously achieves order-optimal communication and computation costs, for a large class of decompositions $\mathbf{X}$. This optimality is derived from our achievability and converse bounds, which reveal -- under reasonable assumptions on the density of $\mathbf{X}$ -- that the optimal scaling of the communication cost takes the form $n/N^{1/d}$, revealing that our design achieves the order-optimal \textit{partitioning gain} that scales as $N^{1/d}$, while also achieving an order-optimal computation cost. Interestingly, this order optimality is achieved in a deterministic manner, and very importantly, it is achieved blindly from $\mathbf{X}$, therefore enabling multiple desired functions to be computed without reshuffling files.