Approximate Gradient Coding via Sparse Random Graphs

TL;DR

This paper introduces gradient coding methods using sparse graphs that enable faster, approximate distributed computation, improving robustness to slow nodes with minimal accuracy loss.

Contribution

It proposes a novel class of gradient codes based on sparse random graphs that allow for efficient, approximate solutions in distributed systems.

Findings

Significant increase in robustness to stragglers with minimal accuracy loss

Sparse graph-based codes enable faster distributed gradient computation

Approximate solutions are sufficient for robust model training

Abstract

Distributed algorithms are often beset by the straggler effect, where the slowest compute nodes in the system dictate the overall running time. Coding-theoretic techniques have been recently proposed to mitigate stragglers via algorithmic redundancy. Prior work in coded computation and gradient coding has mainly focused on exact recovery of the desired output. However, slightly inexact solutions can be acceptable in applications that are robust to noise, such as model training via gradient-based algorithms. In this work, we present computationally simple gradient codes based on sparse graphs that guarantee fast and approximately accurate distributed computation. We demonstrate that sacrificing a small amount of accuracy can significantly increase algorithmic robustness to stragglers.