BlackboxBench: A Comprehensive Benchmark of Black-box Adversarial Attacks
Meixi Zheng, Xuanchen Yan, Zihao Zhu, Hongrui Chen, Baoyuan Wu

TL;DR
BlackboxBench provides a comprehensive, modular benchmark for evaluating and comparing 59 black-box adversarial attack algorithms across multiple models and datasets, facilitating progress analysis in this field.
Contribution
It introduces a unified codebase with extensive evaluations and analysis tools for black-box adversarial attacks, addressing the lack of thorough comparative studies.
Findings
Evaluation of 59 attack algorithms across models and datasets
Identification of strengths and weaknesses of different attack methods
Insights into attack effectiveness and efficiency trade-offs
Abstract
Adversarial examples are well-known tools to evaluate the vulnerability of deep neural networks (DNNs). Although lots of adversarial attack algorithms have been developed, it's still challenging in the practical scenario that the model's parameters and architectures are inaccessible to the attacker/evaluator, i.e., black-box adversarial attacks. Due to the practical importance, there has been rapid progress from recent algorithms, reflected by the quick increase in attack success rate and quick decrease in query numbers to the target model. However, there lacks thorough evaluations and comparisons among these algorithms, causing difficulties in tracking the real progress, analyzing advantages and disadvantages of different technical routes, as well as designing future development roadmap of this field. Thus, we aim at building a comprehensive benchmark of black-box adversarial attacks,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Anomaly Detection Techniques and Applications · Advanced Neural Network Applications
