Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction

TL;DR

This paper introduces SADCA, a novel attack method that enhances the transferability of adversarial examples in vision-language models by using semantic augmentation and dynamic contrastive interactions, leading to more effective cross-model attacks.

Contribution

The paper proposes SADCA, a semantic-augmented dynamic contrastive attack that improves transferability of adversarial examples in vision-language pre-training models through progressive, semantically guided perturbations.

Findings

SADCA outperforms existing attack methods in transferability across multiple datasets and models.

Semantic augmentation increases diversity and generalization of adversarial examples.

Dynamic contrastive interactions reinforce semantic inconsistency, enhancing attack effectiveness.

Abstract

With the rapid advancement and widespread application of vision-language pre-training (VLP) models, their vulnerability to adversarial attacks has become a critical concern. In general, the adversarial examples can typically be designed to exhibit transferable power, attacking not only different models but also across diverse tasks. However, existing attacks on language-vision models mainly rely on static cross-modal interactions and focus solely on disrupting positive image-text pairs, resulting in limited cross-modal disruption and poor transferability. To address this issue, we propose a Semantic-Augmented Dynamic Contrastive Attack (SADCA) that enhances adversarial transferability through progressive and semantically guided perturbation. SADCA progressively disrupts cross-modal alignment through dynamic interactions between adversarial images and texts. This is accomplished by SADCA establishing a contrastive learning mechanism involving adversarial, positive and negative samples, to reinforce the semantic inconsistency of the obtained perturbations. Moreover, we empirically find that input transformations commonly used in traditional transfer-based attacks also benefit VLPs, which motivates a semantic augmentation module that increases the diversity and generalization of adversarial examples. Extensive experiments on multiple datasets and models demonstrate that SADCA significantly improves adversarial transferability and consistently surpasses state-of-the-art methods. The code is released at https://github.com/LiYuanBoJNU/SADCA.