SignSGD: Fault-Tolerance to Blind and Byzantine Adversaries

TL;DR

This paper introduces a robust version of SignSGD that maintains convergence in distributed learning environments even when facing Byzantine adversaries, supported by theoretical proofs and empirical experiments.

Contribution

The paper develops a convergence upper bound for SignSGD under Byzantine faults, demonstrating its robustness compared to standard distributed SGD.

Findings

SignSGD converges despite Byzantine adversaries.

Empirical results support the theoretical robustness of SignSGD.

Code and experiments are publicly available for reproducibility.

Abstract

Distributed learning has become a necessity for training ever-growing models by sharing calculation among several devices. However, some of the devices can be faulty, deliberately or not, preventing the proper convergence. As a matter of fact, the baseline distributed SGD algorithm does not converge in the presence of one Byzantine adversary. In this article we focus on the more robust SignSGD algorithm derived from SGD. We provide an upper bound for the convergence rate of SignSGD proving that this new version is robust to Byzantine adversaries. We implemented SignSGD along with Byzantine strategies attempting to crush the learning process. Therefore, we provide empirical observations from our experiments to support our theory. Our code is available on GitHub https://github.com/jasonakoun/signsgd-fault-tolerance and our experiments are reproducible by using the provided parameters.