Accurate, reliable and fast robustness evaluation

TL;DR

This paper introduces a new set of gradient-based adversarial attacks that are more reliable, efficient, and adaptable, significantly improving the evaluation of neural network robustness across various models and scenarios.

Contribution

Develops a novel gradient-based attack method that is more reliable, query-efficient, adaptable, and requires minimal hyperparameter tuning for robustness evaluation.

Findings

More reliable against gradient-masking

Outperforms existing gradient-based attacks in efficiency

Validated across six diverse models and multiple norms

Abstract

Throughout the past five years, the susceptibility of neural networks to minimal adversarial perturbations has moved from a peculiar phenomenon to a core issue in Deep Learning. Despite much attention, however, progress towards more robust models is significantly impaired by the difficulty of evaluating the robustness of neural network models. Today's methods are either fast but brittle (gradient-based attacks), or they are fairly reliable but slow (score- and decision-based attacks). We here develop a new set of gradient-based adversarial attacks which (a) are more reliable in the face of gradient-masking than other gradient-based attacks, (b) perform better and are more query efficient than current state-of-the-art gradient-based attacks, (c) can be flexibly adapted to a wide range of adversarial criteria and (d) require virtually no hyperparameter tuning. These findings are carefully validated across a diverse set of six different models and hold for L0, L1, L2 and Linf in both targeted as well as untargeted scenarios. Implementations will soon be available in all major toolboxes (Foolbox, CleverHans and ART). We hope that this class of attacks will make robustness evaluations easier and more reliable, thus contributing to more signal in the search for more robust machine learning models.