Accelerating Byzantine-Robust Distributed Learning with Compressed Communication via Double Momentum and Variance Reduction

TL;DR

This paper introduces Byz-DM21, a communication-efficient, Byzantine-robust distributed learning algorithm with a novel double-momentum gradient estimator, achieving faster convergence and reduced variance.

Contribution

It proposes a new gradient estimator with double-momentum for Byzantine robustness and communication efficiency, along with a variance reduction variant for improved convergence.

Findings

Byz-DM21 converges in $ ilde{O}(rac{1}{ ext{epsilon}^4})$ iterations.

Byz-VR-DM21 achieves $ ilde{O}(rac{1}{ ext{epsilon}^3})$ convergence with variance reduction.

Numerical experiments confirm the effectiveness of the proposed algorithms.

Abstract

In collaborative and distributed learning, Byzantine robustness reflects a major facet of optimization algorithms. Such distributed algorithms are often accompanied by transmitting a large number of parameters, so communication compression is essential for an effective solution. In this paper, we propose Byz-DM21, a novel Byzantine-robust and communication-efficient stochastic distributed learning algorithm. Our key innovation is a novel gradient estimator based on a double-momentum mechanism, integrating recent advancements in error feedback techniques. Using this estimator, we design both standard and accelerated algorithms that eliminate the need for large batch sizes while maintaining robustness against Byzantine workers. We prove that the Byz-DM21 algorithm has a smaller neighborhood size and converges to $\varepsilon$-stationary points in $\mathcal{O}(\varepsilon^{-4})$ iterations. To further enhance efficiency, we introduce a distributed variant called Byz-VR-DM21, which incorporates local variance reduction at each node to progressively eliminate variance from random approximations. We show that Byz-VR-DM21 provably converges to $\varepsilon$-stationary points in $\mathcal{O}(\varepsilon^{-3 })$ iterations. Additionally, we extend our results to the case where the functions satisfy the Polyak-{\L}ojasiewicz condition. Finally, numerical experiments demonstrate the effectiveness of the proposed method.