Computation Scheduling for Distributed Machine Learning with Straggling Workers

TL;DR

This paper develops and analyzes scheduling schemes for distributed machine learning that reduce completion time by effectively managing straggling workers and leveraging redundancy, with theoretical and experimental validation.

Contribution

It introduces two novel scheduling schemes for distributed learning that optimize task assignment and order, achieving near-optimal completion times under random delays.

Findings

Proposed schemes significantly reduce average completion time.

Experimental results on Amazon EC2 validate improvements over existing methods.

The gap between schemes and the theoretical lower bound is small.

Abstract

We study scheduling of computation tasks across n workers in a large scale distributed learning problem with the help of a master. Computation and communication delays are assumed to be random, and redundant computations are assigned to workers in order to tolerate stragglers. We consider sequential computation of tasks assigned to a worker, while the result of each computation is sent to the master right after its completion. Each computation round, which can model an iteration of the stochastic gradient descent (SGD) algorithm, is completed once the master receives k distinct computations, referred to as the computation target. Our goal is to characterize the average completion time as a function of the computation load, which denotes the portion of the dataset available at each worker, and the computation target. We propose two computation scheduling schemes that specify the tasks assigned to each worker, as well as their computation schedule, i.e., the order of execution. Assuming a general statistical model for computation and communication delays, we derive the average completion time of the proposed schemes. We also establish a lower bound on the minimum average completion time by assuming prior knowledge of the random delays. Experimental results carried out on Amazon EC2 cluster show a significant reduction in the average completion time over existing coded and uncoded computing schemes. It is also shown numerically that the gap between the proposed scheme and the lower bound is relatively small, confirming the efficiency of the proposed scheduling design.