A Dynamic Weighting Strategy to Mitigate Worker Node Failure in Distributed Deep Learning

TL;DR

This paper introduces a dynamic weighting strategy to address worker node failures in distributed deep learning, improving training efficiency and convergence by mitigating straggler effects.

Contribution

It proposes a novel dynamic weighting approach that enhances robustness and performance in distributed deep learning systems facing node failures.

Findings

Improved convergence rates with the proposed strategy

Enhanced training efficiency in distributed systems

Better test performance under node failure conditions

Abstract

The increasing complexity of deep learning models and the demand for processing vast amounts of data make the utilization of large-scale distributed systems for efficient training essential. These systems, however, face significant challenges such as communication overhead, hardware limitations, and node failure. This paper investigates various optimization techniques in distributed deep learning, including Elastic Averaging SGD (EASGD) and the second-order method AdaHessian. We propose a dynamic weighting strategy to mitigate the problem of straggler nodes due to failure, enhancing the performance and efficiency of the overall training process. We conduct experiments with different numbers of workers and communication periods to demonstrate improved convergence rates and test performance using our strategy.