DRACO: Byzantine-resilient Distributed Training via Redundant Gradients

TL;DR

DRACO is a scalable, coding-theory-based framework for robust distributed training that effectively mitigates malicious node updates while maintaining model accuracy and significantly improving speed over median-based methods.

Contribution

DRACO introduces a novel coding-theory approach for Byzantine resilience in distributed training, providing robustness guarantees without sacrificing efficiency.

Findings

DRACO is several times faster than median-based robust training methods.

It maintains model accuracy comparable to adversary-free training.

DRACO offers problem-independent robustness guarantees.

Abstract

Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i.e., nodes that use malicious updates to corrupt the global model stored at a parameter server (PS). To guarantee some form of robustness, recent work suggests using variants of the geometric median as an aggregation rule, in place of gradient averaging. Unfortunately, median-based rules can incur a prohibitive computational overhead in large-scale settings, and their convergence guarantees often require strong assumptions. In this work, we present DRACO, a scalable framework for robust distributed training that uses ideas from coding theory. In DRACO, each compute node evaluates redundant gradients that are used by the parameter server to eliminate the effects of adversarial updates. DRACO comes with problem-independent robustness guarantees, and the model that it trains is identical to the one trained in the adversary-free setup. We provide extensive experiments on real datasets and distributed setups across a variety of large-scale models, where we show that DRACO is several times, to orders of magnitude faster than median-based approaches.