MindFlayer SGD: Efficient Parallel SGD in the Presence of Heterogeneous and Random Worker Compute Times

TL;DR

MindFlayer SGD is a new parallel stochastic gradient descent algorithm designed to efficiently handle heterogeneous and stochastic worker compute times, improving performance in distributed nonconvex optimization tasks.

Contribution

We introduce MindFlayer SGD, a novel method that effectively manages random and heterogeneous compute delays in distributed SGD, with theoretical and empirical validation.

Findings

Outperforms existing baselines in environments with heavy-tailed noise.

Demonstrates robustness and scalability in large-scale distributed learning.

Maintains efficiency despite stochastic and heterogeneous worker compute times.

Abstract

We investigate the problem of minimizing the expectation of smooth nonconvex functions in a distributed setting with multiple parallel workers that are able to compute stochastic gradients. A significant challenge in this context is the presence of arbitrarily heterogeneous and stochastic compute times among workers, which can severely degrade the performance of existing parallel stochastic gradient descent (SGD) methods. While some parallel SGD algorithms achieve optimal performance under deterministic but heterogeneous delays, their effectiveness diminishes when compute times are random - a scenario not explicitly addressed in their design. To bridge this gap, we introduce MindFlayer SGD, a novel parallel SGD method specifically designed to handle stochastic and heterogeneous compute times. Through theoretical analysis and empirical evaluation, we demonstrate that MindFlayer SGD consistently outperforms existing baselines, particularly in environments with heavy-tailed noise. Our results highlight its robustness and scalability, making it a compelling choice for large-scale distributed learning tasks.