Optimization-based Block Coordinate Gradient Coding

TL;DR

This paper introduces an optimal gradient coding scheme with diverse redundancies across gradient coordinates, improving distributed machine learning efficiency by minimizing expected runtime under partial straggler models.

Contribution

It proposes a novel block coordinate gradient coding method that optimizes redundancy distribution across gradient coordinates, a first in this research area.

Findings

Optimal redundancy levels for different straggler tolerances identified.

Two low-complexity approximate solutions with closed-form expressions developed.

Expected runtime gaps between solutions and optimal are sub-linear in the number of workers.

Abstract

Existing gradient coding schemes introduce identical redundancy across the coordinates of gradients and hence cannot fully utilize the computation results from partial stragglers. This motivates the introduction of diverse redundancies across the coordinates of gradients. This paper considers a distributed computation system consisting of one master and $N$ workers characterized by a general partial straggler model and focuses on solving a general large-scale machine learning problem with $L$ model parameters. We show that it is sufficient to provide at most $N$ levels of redundancies for tolerating $0, 1,\cdots, N-1$ stragglers, respectively. Consequently, we propose an optimal block coordinate gradient coding scheme based on a stochastic optimization problem that optimizes the partition of the $L$ coordinates into $N$ blocks, each with identical redundancy, to minimize the expected overall runtime for collaboratively computing the gradient. We obtain an optimal solution using a stochastic projected subgradient method and propose two low-complexity approximate solutions with closed-from expressions, for the stochastic optimization problem. We also show that under a shifted-exponential distribution, for any $L$, the expected overall runtimes of the two approximate solutions and the minimum overall runtime have sub-linear multiplicative gaps in $N$. To the best of our knowledge, this is the first work that optimizes the redundancies of gradient coding introduced across the coordinates of gradients.