Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics

TL;DR

This paper demonstrates that adversarially robust CLIP models produce superior, robust perceptual similarity metrics that outperform existing methods, especially under adversarial attacks, and enhance interpretability and related task performance.

Contribution

Introducing adversarially robust CLIP models (R-CLIP) that generate more reliable and interpretable perceptual metrics with improved robustness and applicability to related tasks.

Findings

Robust CLIP models outperform existing perceptual metrics in zero-shot settings.

The robust metrics maintain high accuracy under adversarial attacks.

Enhanced interpretability through feature and text inversion visualizations.

Abstract

Measuring perceptual similarity is a key tool in computer vision. In recent years perceptual metrics based on features extracted from neural networks with large and diverse training sets, e.g. CLIP, have become popular. At the same time, the metrics extracted from features of neural networks are not adversarially robust. In this paper we show that adversarially robust CLIP models, called R-CLIP$_\textrm{F}$, obtained by unsupervised adversarial fine-tuning induce a better and adversarially robust perceptual metric that outperforms existing metrics in a zero-shot setting, and further matches the performance of state-of-the-art metrics while being robust after fine-tuning. Moreover, our perceptual metric achieves strong performance on related tasks such as robust image-to-image retrieval, which becomes especially relevant when applied to "Not Safe for Work" (NSFW) content detection and dataset filtering. While standard perceptual metrics can be easily attacked by a small perturbation completely degrading NSFW detection, our robust perceptual metric maintains high accuracy under an attack while having similar performance for unperturbed images. Finally, perceptual metrics induced by robust CLIP models have higher interpretability: feature inversion can show which images are considered similar, while text inversion can find what images are associated to a given prompt. This also allows us to visualize the very rich visual concepts learned by a CLIP model, including memorized persons, paintings and complex queries.