dSTAR: Straggler Tolerant and Byzantine Resilient Distributed SGD

TL;DR

dSTAR is a novel distributed SGD method that enhances robustness against stragglers and Byzantine faults by selective gradient aggregation, ensuring reliable convergence and high accuracy in adversarial and slow network conditions.

Contribution

We introduce dSTAR, a lightweight, Byzantine-resilient distributed SGD algorithm that filters gradients based on ensemble median and guarantees linear convergence.

Findings

dSTAR achieves (, f)-Byzantine resilience.

dSTAR maintains high accuracy under Byzantine attacks.

Empirical results outperform existing Byzantine-resilient methods.

Abstract

Distributed model training needs to be adapted to challenges such as the straggler effect and Byzantine attacks. When coordinating the training process with multiple computing nodes, ensuring timely and reliable gradient aggregation amidst network and system malfunctions is essential. To tackle these issues, we propose \textit{dSTAR}, a lightweight and efficient approach for distributed stochastic gradient descent (SGD) that enhances robustness and convergence. \textit{dSTAR} selectively aggregates gradients by collecting updates from the first \(k\) workers to respond, filtering them based on deviations calculated using an ensemble median. This method not only mitigates the impact of stragglers but also fortifies the model against Byzantine adversaries. We theoretically establish that \textit{dSTAR} is (\(\alpha, f\))-Byzantine resilient and achieves a linear convergence rate. Empirical evaluations across various scenarios demonstrate that \textit{dSTAR} consistently maintains high accuracy, outperforming other Byzantine-resilient methods that often suffer up to a 40-50\% accuracy drop under attack. Our results highlight \textit{dSTAR} as a robust solution for training models in distributed environments prone to both straggler delays and Byzantine faults.