Speeding Up Distributed Gradient Descent by Utilizing Non-persistent Stragglers

TL;DR

This paper proposes a method to reduce the average completion time in distributed gradient descent by enabling multiple transmissions per server, including stragglers, thus improving efficiency with a slight increase in communication load.

Contribution

It introduces a novel approach allowing multiple transmissions from each server per iteration, including stragglers, to maximize completed computations and reduce overall iteration time.

Findings

Average completion time per iteration is significantly reduced.

Multiple transmissions improve utilization of straggling servers.

Slight increase in communication load yields substantial efficiency gains.

Abstract

Distributed gradient descent (DGD) is an efficient way of implementing gradient descent (GD), especially for large data sets, by dividing the computation tasks into smaller subtasks and assigning to different computing servers (CSs) to be executed in parallel. In standard parallel execution, per-iteration waiting time is limited by the execution time of the straggling servers. Coded DGD techniques have been introduced recently, which can tolerate straggling servers via assigning redundant computation tasks to the CSs. In most of the existing DGD schemes, either with coded computation or coded communication, the non-straggling CSs transmit one message per iteration once they complete all their assigned computation tasks. However, although the straggling servers cannot complete all their assigned tasks, they are often able to complete a certain portion of them. In this paper, we allow multiple transmissions from each CS at each iteration in order to make sure a maximum number of completed computations can be reported to the aggregating server (AS), including the straggling servers. We numerically show that the average completion time per iteration can be reduced significantly by slightly increasing the communication load per server.