Blades: A Unified Benchmark Suite for Byzantine Attacks and Defenses in Federated Learning

TL;DR

Blades is a comprehensive benchmark suite designed to evaluate Byzantine attack and defense strategies in federated learning, enabling fair comparison and fostering advancements in robust FL methods.

Contribution

This paper introduces Blades, the first scalable and extensible benchmark suite for systematic evaluation of Byzantine attacks and defenses in federated learning.

Findings

Re-evaluated attacks and defenses across diverse configurations

Identified limitations in existing methods through extensive testing

Provided new insights into FL robustness and vulnerabilities

Abstract

Federated learning (FL) facilitates distributed training across different IoT and edge devices, safeguarding the privacy of their data. The inherent distributed structure of FL introduces vulnerabilities, especially from adversarial devices aiming to skew local updates to their advantage. Despite the plethora of research focusing on Byzantine-resilient FL, the academic community has yet to establish a comprehensive benchmark suite, pivotal for impartial assessment and comparison of different techniques. This paper presents Blades, a scalable, extensible, and easily configurable benchmark suite that supports researchers and developers in efficiently implementing and validating novel strategies against baseline algorithms in Byzantine-resilient FL. Blades contains built-in implementations of representative attack and defense strategies and offers a user-friendly interface that seamlessly integrates new ideas. Using Blades, we re-evaluate representative attacks and defenses on wide-ranging experimental configurations (approximately 1,500 trials in total). Through our extensive experiments, we gained new insights into FL robustness and highlighted previously overlooked limitations due to the absence of thorough evaluations and comparisons of baselines under various attack settings.