Generalized Byzantine-tolerant SGD

TL;DR

This paper introduces three new robust aggregation rules for distributed SGD that withstand Byzantine failures, demonstrating improved resilience and performance in adversarial scenarios.

Contribution

The paper proposes three novel aggregation rules for Byzantine-tolerant distributed SGD and proves their resilience under a general Byzantine failure model.

Findings

Proposed aggregation rules outperform existing methods in realistic scenarios.

The rules are proven to be Byzantine resilient.

Empirical results confirm improved robustness and efficiency.

Abstract

We propose three new robust aggregation rules for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience properties of these aggregation rules. Empirical analysis shows that the proposed techniques outperform current approaches for realistic use cases and Byzantine attack scenarios.