MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Models

TL;DR

This paper introduces MoAPT, a novel adversarial prompt tuning method for vision-language models that improves robustness against various adversarial attacks by learning mixture prompts and using a conditional weight router.

Contribution

Proposes MoAPT, a mixture of adversarial prompt tuning with a conditional weight router, enhancing VLM robustness against diverse adversarial attacks.

Findings

MoAPT outperforms state-of-the-art methods in robustness across 11 datasets.

Increasing the number of learned prompts improves adversarial robustness.

Sample-specific prompt weighting enhances alignment with adversarial images.

Abstract

Large pre-trained Vision Language Models (VLMs) demonstrate excellent generalization capabilities but remain highly susceptible to adversarial examples, posing potential security risks. To improve the robustness of VLMs against adversarial examples, adversarial prompt tuning methods are proposed to align the text feature with the adversarial image feature without changing model parameters. However, when facing various adversarial attacks, a single learnable text prompt has insufficient generalization to align well with all adversarial image features, which ultimately results in overfitting. To address the above challenge, in this paper, we empirically find that increasing the number of learned prompts yields greater robustness improvements than simply extending the length of a single prompt. Building on this observation, we propose an adversarial tuning method named \textbf{Mixture of Adversarial Prompt Tuning (MoAPT)} to enhance the generalization against various adversarial attacks for VLMs. MoAPT aims to learn mixture text prompts to obtain more robust text features. To further enhance the adaptability, we propose a conditional weight router based on the adversarial images to predict the mixture weights of multiple learned prompts, which helps obtain sample-specific mixture text features aligning with different adversarial image features. Extensive experiments across 11 datasets under different settings show that our method can achieve better adversarial robustness than state-of-the-art approaches.