Decoding FL Defenses: Systemization, Pitfalls, and Remedies
Momin Ahmad Khan, Virat Shejwalkar, Yasra Chandio, Amir Houmansadr,, Fatima Muhammad Anwar

TL;DR
This paper systematically analyzes federated learning defenses, revealing common evaluation pitfalls, and offers guidelines to improve the reliability of robustness assessments in FL security research.
Contribution
It introduces a comprehensive systemization of FL defenses, identifies prevalent evaluation pitfalls, and proposes actionable recommendations to enhance defense assessment accuracy.
Findings
30% of studies use only MNIST dataset
40% employ simplistic attacks, risking false robustness claims
Critical reevaluation shows pitfalls lead to incorrect robustness conclusions
Abstract
While the community has designed various defenses to counter the threat of poisoning attacks in Federated Learning (FL), there are no guidelines for evaluating these defenses. These defenses are prone to subtle pitfalls in their experimental setups that lead to a false sense of security, rendering them unsuitable for practical deployment. In this paper, we systematically understand, identify, and provide a better approach to address these challenges. First, we design a comprehensive systemization of FL defenses along three dimensions: i) how client updates are processed, ii) what the server knows, and iii) at what stage the defense is applied. Next, we thoroughly survey 50 top-tier defense papers and identify the commonly used components in their evaluation setups. Based on this survey, we uncover six distinct pitfalls and study their prevalence. For example, we discover that around 30%…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCryptographic Implementations and Security · Security and Verification in Computing · Advanced Malware Detection Techniques
