Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models

TL;DR

This paper introduces a novel attack method called Set-level Guidance Attack (SGA) that significantly improves the transferability of adversarial examples across vision-language pre-training models by leveraging cross-modal interactions and alignment-preserving augmentations.

Contribution

The paper presents the first study on adversarial transferability in VLP models and proposes SGA, a highly transferable attack method that outperforms existing approaches in cross-model attacks.

Findings

SGA significantly increases transfer attack success rates.

Cross-modal interactions are crucial for transferability.

SGA outperforms state-of-the-art methods in experiments.

Abstract

Vision-language pre-training (VLP) models have shown vulnerability to adversarial examples in multimodal tasks. Furthermore, malicious adversaries can be deliberately transferred to attack other black-box models. However, existing work has mainly focused on investigating white-box attacks. In this paper, we present the first study to investigate the adversarial transferability of recent VLP models. We observe that existing methods exhibit much lower transferability, compared to the strong attack performance in white-box settings. The transferability degradation is partly caused by the under-utilization of cross-modal interactions. Particularly, unlike unimodal learning, VLP models rely heavily on cross-modal interactions and the multimodal alignments are many-to-many, e.g., an image can be described in various natural languages. To this end, we propose a highly transferable Set-level Guidance Attack (SGA) that thoroughly leverages modality interactions and incorporates alignment-preserving augmentation with cross-modal guidance. Experimental results demonstrate that SGA could generate adversarial examples that can strongly transfer across different VLP models on multiple downstream vision-language tasks. On image-text retrieval, SGA significantly enhances the attack success rate for transfer attacks from ALBEF to TCL by a large margin (at least 9.78% and up to 30.21%), compared to the state-of-the-art.