TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models

TL;DR

This paper introduces TAPT, a test-time adversarial prompt tuning method that significantly improves the robustness of CLIP against visual adversarial attacks without sacrificing much accuracy on clean data.

Contribution

TAPT is a novel unsupervised, test-time defense approach that learns bimodal prompts to enhance CLIP's robustness against adversarial perturbations.

Findings

Increases CLIP's adversarial robustness by at least 48.9% against AutoAttack.

Maintains high performance on clean examples.

Outperforms existing adversarial prompt tuning methods.

Abstract

Large pre-trained Vision-Language Models (VLMs) such as CLIP have demonstrated excellent zero-shot generalizability across various downstream tasks. However, recent studies have shown that the inference performance of CLIP can be greatly degraded by small adversarial perturbations, especially its visual modality, posing significant safety threats. To mitigate this vulnerability, in this paper, we propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks. TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP. Specifically, it is an unsupervised method that optimizes the defensive prompts for each test sample by minimizing a multi-view entropy and aligning adversarial-clean distributions. We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets, demonstrating that it enhances the zero-shot adversarial robustness of the original CLIP by at least 48.9% against AutoAttack (AA), while largely maintaining performance on clean examples. Moreover, TAPT outperforms existing adversarial prompt tuning methods across various backbones, achieving an average robustness improvement of at least 36.6%.