Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks

TL;DR

This paper introduces a novel attack method, PMA, for evaluating adversarial robustness of image classifiers, and conducts the first million-scale robustness evaluation revealing significant insights into model vulnerabilities.

Contribution

It proposes the Probability Margin Attack (PMA), a new individual attack method, and performs the first large-scale million-image adversarial robustness evaluation.

Findings

PMA outperforms existing individual attack methods.

Ensemble attacks can be more effective and efficient.

Large-scale evaluation reveals robustness gaps in models.

Abstract

As deep learning models are increasingly deployed in safety-critical applications, evaluating their vulnerabilities to adversarial perturbations is essential for ensuring their reliability and trustworthiness. Over the past decade, a large number of white-box adversarial robustness evaluation methods (i.e., attacks) have been proposed, ranging from single-step to multi-step methods and from individual to ensemble methods. Despite these advances, challenges remain in conducting meaningful and comprehensive robustness evaluations, particularly when it comes to large-scale testing and ensuring evaluations reflect real-world adversarial risks. In this work, we focus on image classification models and propose a novel individual attack method, Probability Margin Attack (PMA), which defines the adversarial margin in the probability space rather than the logits space. We analyze the relationship between PMA and existing cross-entropy or logits-margin-based attacks, and show that PMA can outperform the current state-of-the-art individual methods. Building on PMA, we propose two types of ensemble attacks that balance effectiveness and efficiency. Furthermore, we create a million-scale dataset, CC1M, derived from the existing CC3M dataset, and use it to conduct the first million-scale white-box adversarial robustness evaluation of adversarially-trained ImageNet models. Our findings provide valuable insights into the robustness gaps between individual versus ensemble attacks and small-scale versus million-scale evaluations.