Revisiting the Robust Generalization of Adversarial Prompt Tuning

TL;DR

This paper introduces CAPT, an adaptive prompt tuning framework that improves adversarial robustness and generalization of CLIP models across multiple datasets by balancing consistency between clean and adversarial inputs.

Contribution

The paper proposes a novel adaptive consistency-guided adversarial prompt tuning method that enhances multi-modal feature alignment and robustness of pre-trained vision-language models.

Findings

CAPT outperforms existing methods on 14 datasets.

It maintains high accuracy on clean data while improving adversarial robustness.

CAPT shows strong generalization under distribution shifts.

Abstract

Understanding the vulnerability of large-scale pre-trained vision-language models like CLIP against adversarial attacks is key to ensuring zero-shot generalization capacity on various downstream tasks. State-of-the-art defense mechanisms generally adopt prompt learning strategies for adversarial fine-tuning to improve the adversarial robustness of the pre-trained model while keeping the efficiency of adapting to downstream tasks. Such a setup leads to the problem of over-fitting which impedes further improvement of the model's generalization capacity on both clean and adversarial examples. In this work, we propose an adaptive Consistency-guided Adversarial Prompt Tuning (i.e., CAPT) framework that utilizes multi-modal prompt learning to enhance the alignment of image and text features for adversarial examples and leverage the strong generalization of pre-trained CLIP to guide the model-enhancing its robust generalization on adversarial examples while maintaining its accuracy on clean ones. We also design a novel adaptive consistency objective function to balance the consistency of adversarial inputs and clean inputs between the fine-tuning model and the pre-trained model. We conduct extensive experiments across 14 datasets and 4 data sparsity schemes (from 1-shot to full training data settings) to show the superiority of CAPT over other state-of-the-art adaption methods. CAPT demonstrated excellent performance in terms of the in-distribution performance and the generalization under input distribution shift and across datasets.