Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Models

TL;DR

This paper introduces Feedback-based Modal Mutual Search (FMMS), a novel black-box attack method that uses target model feedback and a modal mutual loss to generate more transferable adversarial examples for vision-language models.

Contribution

It proposes a new attack paradigm leveraging target model feedback and a modal mutual loss to improve transferability of adversarial examples in vision-language models.

Findings

FMMS significantly outperforms state-of-the-art baselines.

It effectively explores multi-modality adversarial boundaries.

The method enhances attack success rates on image-text matching tasks.

Abstract

Although vision-language pre-training (VLP) models have achieved remarkable progress on cross-modal tasks, they remain vulnerable to adversarial attacks. Using data augmentation and cross-modal interactions to generate transferable adversarial examples on surrogate models, transfer-based black-box attacks have become the mainstream methods in attacking VLP models, as they are more practical in real-world scenarios. However, their transferability may be limited due to the differences on feature representation across different models. To this end, we propose a new attack paradigm called Feedback-based Modal Mutual Search (FMMS). FMMS introduces a novel modal mutual loss (MML), aiming to push away the matched image-text pairs while randomly drawing mismatched pairs closer in feature space, guiding the update directions of the adversarial examples. Additionally, FMMS leverages the target model feedback to iteratively refine adversarial examples, driving them into the adversarial region. To our knowledge, this is the first work to exploit target model feedback to explore multi-modality adversarial boundaries. Extensive empirical evaluations on Flickr30K and MSCOCO datasets for image-text matching tasks show that FMMS significantly outperforms the state-of-the-art baselines.