Reconciling Communication Compression and Byzantine-Robustness in Distributed Learning

TL;DR

This paper introduces RoSDHB, a new distributed learning algorithm that balances communication efficiency and Byzantine fault tolerance, improving robustness and reducing communication costs compared to previous methods.

Contribution

RoSDHB combines classical momentum with coordinated compression, offering better robustness and efficiency under milder assumptions than prior state-of-the-art algorithms.

Findings

RoSDHB matches convergence guarantees of Byz-DASHA-PAGE.

RoSDHB demonstrates stronger robustness in experiments.

RoSDHB achieves significant communication savings.

Abstract

Distributed learning enables scalable model training over decentralized data, but remains hindered by Byzantine faults and high communication costs. While both challenges have been studied extensively in isolation, their interplay has received limited attention. Prior work has shown that naively combining communication compression with Byzantine-robust aggregation can severely weaken resilience to faulty nodes. The current state-of-the-art, Byz-DASHA-PAGE, leverages a momentum-based variance reduction scheme to counteract the negative effect of compression noise on Byzantine robustness. In this work, we introduce RoSDHB, a new algorithm that integrates classical Polyak momentum with a coordinated compression strategy. Theoretically, RoSDHB matches the convergence guarantees of Byz-DASHA-PAGE under the standard $(G,B)$-gradient dissimilarity model, while relying on milder assumptions and requiring less memory and communication per client. Empirically, RoSDHB demonstrates stronger robustness while achieving substantial communication savings compared to Byz-DASHA-PAGE.