On the Efficacy of Metrics to Describe Adversarial Attacks

TL;DR

This paper investigates the effectiveness of traditional L-norm metrics in evaluating adversarial attacks and finds that image quality metrics can better predict attack detectability and classification, suggesting a need to revise evaluation methods.

Contribution

The study introduces the use of image quality metrics to assess adversarial attack detectability, challenging the reliance on L-norms alone for evaluation.

Findings

L-norms are rarely the best sole metric for attack evaluation

Image quality metrics can predict detector responses with high accuracy (~0.94)

Metrics can classify attacks based on perturbation similarity and detectability

Abstract

Adversarial defenses are naturally evaluated on their ability to tolerate adversarial attacks. To test defenses, diverse adversarial attacks are crafted, that are usually described in terms of their evading capability and the L0, L1, L2, and Linf norms. We question if the evading capability and L-norms are the most effective information to claim that defenses have been tested against a representative attack set. To this extent, we select image quality metrics from the state of the art and search correlations between image perturbation and detectability. We observe that computing L-norms alone is rarely the preferable solution. We observe a strong correlation between the identified metrics computed on an adversarial image and the output of a detector on such an image, to the extent that they can predict the response of a detector with approximately 0.94 accuracy. Further, we observe that metrics can classify attacks based on similar perturbations and similar detectability. This suggests a possible review of the approach to evaluate detectors, where additional metrics are included to assure that a representative attack dataset is selected.