Semantic-aware Adversarial Fine-tuning for CLIP

TL;DR

This paper introduces Semantic-aware Adversarial Fine-tuning (SAFT), a novel method that enhances CLIP's zero-shot adversarial robustness by generating semantically enriched adversarial examples and fine-tuning with them.

Contribution

The paper proposes a semantic-ensemble attack and SAFT, a new fine-tuning approach that improves CLIP's robustness against adversarial attacks using semantic-aware examples.

Findings

SAFT outperforms existing methods in robustness across 16 datasets.

Semantic-aware AEs are more effective in fooling CLIP than traditional cosine similarity-based AEs.

Extensive experiments validate the effectiveness of SAFT in adversarial robustness.

Abstract

Recent studies have shown that CLIP model's adversarial robustness in zero-shot classification tasks can be enhanced by adversarially fine-tuning its image encoder with adversarial examples (AEs), which are generated by minimizing the cosine similarity between images and a hand-crafted template (e.g., ''A photo of a {label}''). However, it has been shown that the cosine similarity between a single image and a single hand-crafted template is insufficient to measure the similarity for image-text pairs. Building on this, in this paper, we find that the AEs generated using cosine similarity may fail to fool CLIP when the similarity metric is replaced with semantically enriched alternatives, making the image encoder fine-tuned with these AEs less robust. To overcome this issue, we first propose a semantic-ensemble attack to generate semantic-aware AEs by minimizing the average similarity between the original image and an ensemble of refined textual descriptions. These descriptions are initially generated by a foundation model to capture core semantic features beyond hand-crafted templates and are then refined to reduce hallucinations. To this end, we propose Semantic-aware Adversarial Fine-Tuning (SAFT), which fine-tunes CLIP's image encoder with semantic-aware AEs. Extensive experiments show that SAFT outperforms current methods, achieving substantial improvements in zero-shot adversarial robustness across 16 datasets. Our code is available at: https://github.com/tmlr-group/SAFT.