Communication-Efficient and Byzantine-Robust Distributed Learning with Error Feedback

TL;DR

This paper introduces a communication-efficient distributed learning algorithm that is robust against Byzantine failures, using gradient norm thresholding and gradient compression techniques, with proven error bounds and empirical validation.

Contribution

It proposes a novel Byzantine-robust distributed gradient descent method combining gradient norm thresholding with compressed gradients, achieving optimal error rates and improved convergence with error feedback.

Findings

Error rate matches more complex schemes

Compression does not affect convergence in certain regimes

Error feedback improves statistical error rate

Abstract

We develop a communication-efficient distributed learning algorithm that is robust against Byzantine worker machines. We propose and analyze a distributed gradient-descent algorithm that performs a simple thresholding based on gradient norms to mitigate Byzantine failures. We show the (statistical) error-rate of our algorithm matches that of Yin et al.~\cite{dong}, which uses more complicated schemes (coordinate-wise median, trimmed mean). Furthermore, for communication efficiency, we consider a generic class of $\delta$-approximate compressors from Karimireddi et al.~\cite{errorfeed} that encompasses sign-based compressors and top-$k$ sparsification. Our algorithm uses compressed gradients and gradient norms for aggregation and Byzantine removal respectively. We establish the statistical error rate for non-convex smooth loss functions. We show that, in certain range of the compression factor $\delta$, the (order-wise) rate of convergence is not affected by the compression operation. Moreover, we analyze the compressed gradient descent algorithm with error feedback (proposed in \cite{errorfeed}) in a distributed setting and in the presence of Byzantine worker machines. We show that exploiting error feedback improves the statistical error rate. Finally, we experimentally validate our results and show good performance in convergence for convex (least-square regression) and non-convex (neural network training) problems.