Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation

TL;DR

This paper demonstrates that existing Byzantine-tolerant aggregation methods for distributed SGD, such as coordinate-wise median and Krum, can be compromised through inner product manipulation attacks, both theoretically and empirically.

Contribution

It introduces new attack strategies that break two widely used Byzantine-tolerant methods, revealing vulnerabilities in current defenses.

Findings

Coordinate-wise median and Krum can be bypassed using inner product manipulation

Theoretical proofs confirm the effectiveness of the attacks

Empirical experiments validate the attack success in practice

Abstract

Recently, new defense techniques have been developed to tolerate Byzantine failures for distributed machine learning. The Byzantine model captures workers that behave arbitrarily, including malicious and compromised workers. In this paper, we break two prevailing Byzantine-tolerant techniques. Specifically we show robust aggregation methods for synchronous SGD -- coordinate-wise median and Krum -- can be broken using new attack strategies based on inner product manipulation. We prove our results theoretically, as well as show empirical validation.