Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients

TL;DR

Solon is a novel distributed training framework that balances communication efficiency and Byzantine robustness by leveraging gradient redundancy, achieving faster training speeds and resilience against attacks.

Contribution

We introduce Solon, a new algorithmic framework that exploits gradient redundancy to simultaneously improve communication efficiency and Byzantine robustness in distributed training.

Findings

Solon achieves over 10x speedup compared to Bulyan.

Solon is 80% faster than Draco.

Solon maintains convergence under Byzantine attacks that break other methods.

Abstract

There has been a growing need to provide Byzantine-resilience in distributed model training. Existing robust distributed learning algorithms focus on developing sophisticated robust aggregators at the parameter servers, but pay less attention to balancing the communication cost and robustness. In this paper, we propose Solon, an algorithmic framework that exploits gradient redundancy to provide communication efficiency and Byzantine robustness simultaneously. Our theoretical analysis shows a fundamental trade-off among computational load, communication cost, and Byzantine robustness. We also develop a concrete algorithm to achieve the optimal trade-off, borrowing ideas from coding theory and sparse recovery. Empirical experiments on various datasets demonstrate that Solon provides significant speedups over existing methods to achieve the same accuracy, over 10 times faster than Bulyan and 80% faster than Draco. We also show that carefully designed Byzantine attacks break Signum and Bulyan, but do not affect the successful convergence of Solon.