BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning

TL;DR

BackFed provides a standardized benchmark for evaluating backdoor attacks and defenses in federated learning, revealing limitations of current methods and promoting more reliable research progress.

Contribution

We introduce BackFed, a comprehensive benchmark suite that standardizes and stress-tests FL backdoor evaluations, uncovering critical limitations of existing methods.

Findings

Existing attacks and defenses show limited effectiveness in BackFed.

Malicious clients face high costs, vulnerable to time constraints.

Defenses often cause accuracy loss or high overhead.

Abstract

Research on backdoor attacks in Federated Learning (FL) has accelerated in recent years, with new attacks and defenses continually proposed in an escalating arms race. However, the evaluation of these methods remains neither standardized nor reliable. First, there are severe inconsistencies in the evaluation settings across studies, and many rely on unrealistic threat models. Second, our code review uncovers semantic bugs in the official codebases of several attacks that artificially inflate their reported performance. These issues raise fundamental questions about whether current methods are truly effective or simply overfitted to narrow experimental setups. We introduce \textbf{BackFed}, a benchmark designed to standardize and stress-test FL backdoor evaluation by unifying attacks and defenses under a common evaluation framework that mirrors realistic FL deployments. Our benchmark on three representative datasets with three distinct architectures reveals critical limitations of existing methods. Malicious clients often require excessive training time and computation, making them vulnerable to server-enforced time constraints. Meanwhile, several defenses incur severe accuracy degradation or aggregation overhead. Popular defenses and attacks achieve limited performance in our benchmark, which challenges their previous efficacy claims. We establish BackFed as a rigorous and fair evaluation framework that enables more reliable progress in FL backdoor research.