Hierarchical Coded Computation

TL;DR

This paper introduces a hierarchical coded computation approach that leverages partial work from all nodes in distributed systems, significantly reducing expected computation time by optimizing layered erasure codes.

Contribution

It proposes a novel hierarchical coding scheme that exploits partial node work and designs layer-specific codes to improve distributed computation efficiency.

Findings

Achieves 1.5x reduction in expected finishing time.

Design guidelines for optimizing layered erasure codes.

Extends coded computation to utilize partial work by stragglers.

Abstract

Coded computation is a method to mitigate "stragglers" in distributed computing systems through the use of error correction coding that has lately received significant attention. First used in vector-matrix multiplication, the range of application was later extended to include matrix-matrix multiplication, heterogeneous networks, convolution, and approximate computing. A drawback to previous results is they completely ignore work completed by stragglers. While stragglers are slower compute nodes, in many settings the amount of work completed by stragglers can be non-negligible. Thus, in this work, we propose a hierarchical coded computation method that exploits the work completed by all compute nodes. We partition each node's computation into layers of sub-computations such that each layer can be treated as (distinct) erasure channel. We then design different erasure codes for each layer so that all layers have the same failure exponent. We propose design guidelines to optimize parameters of such codes. Numerical results show the proposed scheme has an improvement of a factor of 1.5 in the expected finishing time compared to previous work.