Attacking Perceptual Similarity Metrics

TL;DR

This paper systematically evaluates the robustness of perceptual similarity metrics against imperceptible adversarial attacks, revealing their vulnerability and providing a benchmark for future robustness improvements.

Contribution

It introduces a comprehensive adversarial attack framework against perceptual metrics and benchmarks their robustness, highlighting vulnerabilities and guiding future research.

Findings

All studied metrics are susceptible to common adversarial attacks.

Spatial-transformation attacks transfer effectively across metrics.

Combining attack methods increases transferability and robustness testing.

Abstract

Perceptual similarity metrics have progressively become more correlated with human judgments on perceptual similarity; however, despite recent advances, the addition of an imperceptible distortion can still compromise these metrics. In our study, we systematically examine the robustness of these metrics to imperceptible adversarial perturbations. Following the two-alternative forced-choice experimental design with two distorted images and one reference image, we perturb the distorted image closer to the reference via an adversarial attack until the metric flips its judgment. We first show that all metrics in our study are susceptible to perturbations generated via common adversarial attacks such as FGSM, PGD, and the One-pixel attack. Next, we attack the widely adopted LPIPS metric using spatial-transformation-based adversarial perturbations (stAdv) in a white-box setting to craft adversarial examples that can effectively transfer to other similarity metrics in a black-box setting. We also combine the spatial attack stAdv with PGD ($\ell_\infty$-bounded) attack to increase transferability and use these adversarial examples to benchmark the robustness of both traditional and recently developed metrics. Our benchmark provides a good starting point for discussion and further research on the robustness of metrics to imperceptible adversarial perturbations.