Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Top
Eduard Gorbunov, Samuel Horv\'ath, Peter Richt\'arik, Gauthier Gidel

TL;DR
This paper introduces Byz-VR-MARINA, a Byzantine-tolerant optimization method that leverages variance reduction and communication compression, providing stronger theoretical guarantees and practical efficiency for federated learning scenarios.
Contribution
It presents the first Byzantine-robust method combining variance reduction and compression with tight complexity bounds under weak assumptions.
Findings
Outperforms previous methods in convergence speed.
Supports non-uniform stochastic gradient sampling.
Validated by numerical experiments.
Abstract
Byzantine-robustness has been gaining a lot of attention due to the growth of the interest in collaborative and federated learning. However, many fruitful directions, such as the usage of variance reduction for achieving robustness and communication compression for reducing communication costs, remain weakly explored in the field. This work addresses this gap and proposes Byz-VR-MARINA - a new Byzantine-tolerant method with variance reduction and compression. A key message of our paper is that variance reduction is key to fighting Byzantine workers more effectively. At the same time, communication compression is a bonus that makes the process more communication efficient. We derive theoretical convergence guarantees for Byz-VR-MARINA outperforming previous state-of-the-art for general non-convex and Polyak-Lojasiewicz loss functions. Unlike the concurrent Byzantine-robust methods with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsStochastic Gradient Optimization Techniques · Privacy-Preserving Technologies in Data · Random Matrices and Applications
