Efficient AllReduce with Stragglers

TL;DR

StragglAR is a novel AllReduce algorithm that exploits GPU execution time variation to reduce synchronization delays, achieving significant speedups in distributed machine learning workloads.

Contribution

It introduces a new parallel AllReduce algorithm that leverages natural GPU execution asymmetry to mitigate straggler effects, surpassing bandwidth-efficient methods.

Findings

Achieves 2x theoretical speedup over bandwidth-efficient algorithms.

Provides 25% speedup on an 8-GPU server compared to state-of-the-art methods.

Surpasses the lower bound for bandwidth-optimal AllReduce by exploiting asymmetry.

Abstract

Distributed machine learning workloads use data and tensor parallelism for training and inference, both of which rely on the AllReduce collective to synchronize gradients or activations. However, AllReduce algorithms are delayed by the slowest GPU to reach the synchronization barrier before the collective (i.e., the straggler). To address this challenge, we propose StragglAR: a parallel algorithm for AllReduce that accelerates distributed training and inference by exploiting natural variation in GPU execution times. StragglAR implements a ReduceScatter among the remaining GPUs during the straggler-induced delay, and then executes a novel collective algorithm to complete the AllReduce once the final GPU reaches the synchronization barrier. StragglAR achieves a 2x theoretical speedup over popular bandwidth-efficient algorithms for large GPU clusters, surpassing the lower bound for bandwidth-optimal synchronous AllReduce by leveraging the asymmetry in when GPUs reach the synchronization barrier. On an 8-GPU server, StragglAR provides a 25% speedup over state-of-the-art AllReduce algorithms.