BlackboxBench: A Comprehensive Benchmark of Black-box Adversarial Attacks

TL;DR

BlackboxBench provides a comprehensive, modular benchmark for evaluating and comparing 59 black-box adversarial attack algorithms across multiple models and datasets, facilitating progress analysis in this field.

Contribution

It introduces a unified codebase with extensive evaluations and analysis tools for black-box adversarial attacks, addressing the lack of thorough comparative studies.

Findings

Evaluation of 59 attack algorithms across models and datasets

Identification of strengths and weaknesses of different attack methods

Insights into attack effectiveness and efficiency trade-offs

Abstract

Adversarial examples are well-known tools to evaluate the vulnerability of deep neural networks (DNNs). Although lots of adversarial attack algorithms have been developed, it's still challenging in the practical scenario that the model's parameters and architectures are inaccessible to the attacker/evaluator, i.e., black-box adversarial attacks. Due to the practical importance, there has been rapid progress from recent algorithms, reflected by the quick increase in attack success rate and quick decrease in query numbers to the target model. However, there lacks thorough evaluations and comparisons among these algorithms, causing difficulties in tracking the real progress, analyzing advantages and disadvantages of different technical routes, as well as designing future development roadmap of this field. Thus, we aim at building a comprehensive benchmark of black-box adversarial attacks, called BlackboxBench. It mainly provides: 1) a unified, extensible and modular-based codebase, implementing 29 query-based attack algorithms and 30 transfer-based attack algorithms; 2) comprehensive evaluations: we evaluate the implemented algorithms against several mainstreaming model architectures on 2 widely used datasets (CIFAR-10 and a subset of ImageNet), leading to 14,950 evaluations in total; 3) thorough analysis and new insights, as well analytical tools. The website and source codes of BlackboxBench are available at https://blackboxbenchmark.github.io/ and https://github.com/SCLBD/BlackboxBench/, respectively.