Unified Breakdown Analysis for Byzantine Robust Gossip

TL;DR

This paper proposes a unified framework for Byzantine-robust decentralized learning, introducing a new aggregation rule with near-optimal breakdown point and validating its effectiveness through experiments.

Contribution

It introduces F-RG, a general framework for robust decentralized algorithms, and a new aggregation rule CS+ with near-optimal breakdown point.

Findings

CS+-RG achieves near-optimal robustness against Byzantine failures.

Experimental results show CS+-RG outperforms existing algorithms like NNA.

The paper establishes an upper bound on the number of adversaries tolerated.

Abstract

In decentralized machine learning, different devices communicate in a peer-to-peer manner to collaboratively learn from each other's data. Such approaches are vulnerable to misbehaving (or Byzantine) devices. We introduce F-RG, a general framework for building robust decentralized algorithms with guarantees arising from robust-sum-like aggregation rules F. We then investigate the notion of *breakdown point*, and show an upper bound on the number of adversaries that decentralized algorithms can tolerate. We introduce a practical robust aggregation rule, coined CS+, such that CS+-RG has a near-optimal breakdown. Other choices of aggregation rules lead to existing algorithms such as ClippedGossip or NNA. We give experimental evidence to validate the effectiveness of CS+-RG and highlight the gap with NNA, in particular against a novel attack tailored to decentralized communications.