RobustBench: a standardized adversarial robustness benchmark
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo, Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, Matthias Hein

TL;DR
RobustBench establishes a standardized, comprehensive benchmark for evaluating adversarial robustness in image classification, enabling consistent comparison of models and fostering progress in the field.
Contribution
It introduces a unified evaluation framework using AutoAttack, hosts a leaderboard with 120+ models, and provides open-source tools for robust model analysis and comparison.
Findings
AutoAttack improves robustness evaluation accuracy
Benchmark reveals current state-of-the-art robustness levels
Analysis of robustness impact on various model properties
Abstract
As a research community, we are still lacking a systematic understanding of the progress on adversarial robustness which often makes it hard to identify the most promising ideas in training robust models. A key challenge in benchmarking robustness is that its evaluation is often error-prone leading to robustness overestimation. Our goal is to establish a standardized benchmark of adversarial robustness, which as accurately as possible reflects the robustness of the considered models within a reasonable computational budget. To this end, we start by considering the image classification task and introduce restrictions (possibly loosened in the future) on the allowed models. We evaluate adversarial robustness with AutoAttack, an ensemble of white- and black-box attacks, which was recently shown in a large-scale study to improve almost all robustness evaluations compared to the original…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Cardiac Arrest and Resuscitation
