SoK: The Last Line of Defense: On Backdoor Defense Evaluation
Gorka Abad, Marina Kr\v{c}ek, Stefanos Koffas, Behrad Tajalli, Marco Arazzi, Roberto Ria\~no, Xiaoyun Xu, Zhuoran Liu, Antonino Nocera, Stjepan Picek

TL;DR
This paper systematically reviews and empirically evaluates backdoor defenses in deep learning, highlighting inconsistencies in evaluation practices and proposing standardized guidelines to improve future assessments.
Contribution
It provides a comprehensive meta-analysis of backdoor defense evaluations and offers concrete recommendations for standardizing evaluation methodologies.
Findings
Significant variability in defense effectiveness across different setups
Identified gaps in evaluation practices such as reporting overhead and benign behavior
Proposed guidelines to improve consistency and fairness in future evaluations
Abstract
Backdoor attacks pose a significant threat to deep learning models by implanting hidden vulnerabilities that can be activated by malicious inputs. While numerous defenses have been proposed to mitigate these attacks, the heterogeneous landscape of evaluation methodologies hinders fair comparison between defenses. This work presents a systematic (meta-)analysis of backdoor defenses through a comprehensive literature review and empirical evaluation. We analyzed 183 backdoor defense papers published between 2018 and 2025 across major AI and security venues, examining the properties and evaluation methodologies of these defenses. Our analysis reveals significant inconsistencies in experimental setups, evaluation metrics, and threat model assumptions in the literature. Through extensive experiments involving three datasets (MNIST, CIFAR-100, ImageNet-1K), four model architectures…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Advanced Malware Detection Techniques · Security and Verification in Computing
