Age-Based Coded Computation for Bias Reduction in Distributed Learning

TL;DR

This paper introduces an age-based dynamic encoding framework for coded distributed learning, reducing bias and accelerating convergence by improving the timeliness of partial gradient recovery amidst straggling workers.

Contribution

It proposes a novel age-based dynamic encoding scheme that adapts over time to mitigate bias in partial gradient recovery in distributed learning.

Findings

Reduces bias in gradient estimators

Speeds up convergence in distributed learning

Enhances timeliness of partial computations

Abstract

Coded computation can be used to speed up distributed learning in the presence of straggling workers. Partial recovery of the gradient vector can further reduce the computation time at each iteration; however, this can result in biased estimators, which may slow down convergence, or even cause divergence. Estimator bias will be particularly prevalent when the straggling behavior is correlated over time, which results in the gradient estimators being dominated by a few fast servers. To mitigate biased estimators, we design a $timely$ dynamic encoding framework for partial recovery that includes an ordering operator that changes the codewords and computation orders at workers over time. To regulate the recovery frequencies, we adopt an $age$ metric in the design of the dynamic encoding scheme. We show through numerical results that the proposed dynamic encoding strategy increases the timeliness of the recovered computations, which as a result, reduces the bias in model updates, and accelerates the convergence compared to the conventional static partial recovery schemes.