Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning

TL;DR

This paper presents a new gradient-based multimodal adversarial attack method using contrastive learning to improve transferability of adversarial samples in vision-language models, enhancing robustness evaluation.

Contribution

Introduces a novel multimodal adversarial attack leveraging contrastive learning to generate transferable adversarial samples for VLP models.

Findings

Significant improvement over existing attack methods.

Effective in black-box settings for retrieval and entailment tasks.

Enhances understanding of model robustness against multimodal attacks.

Abstract

The integration of visual and textual data in Vision-Language Pre-training (VLP) models is crucial for enhancing vision-language understanding. However, the adversarial robustness of these models, especially in the alignment of image-text features, has not yet been sufficiently explored. In this paper, we introduce a novel gradient-based multimodal adversarial attack method, underpinned by contrastive learning, to improve the transferability of multimodal adversarial samples in VLP models. This method concurrently generates adversarial texts and images within imperceptive perturbation, employing both image-text and intra-modal contrastive loss. We evaluate the effectiveness of our approach on image-text retrieval and visual entailment tasks, using publicly available datasets in a black-box setting. Extensive experiments indicate a significant advancement over existing single-modal transfer-based adversarial attack methods and current multimodal adversarial attack approaches.