Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack

TL;DR

This paper introduces LSSA, a novel attack method that improves the transferability of adversarial examples in visual-language models by shuffling image blocks and sampling, leading to more effective cross-modal attacks.

Contribution

The paper proposes LSSA, a new attack strategy that enhances adversarial transferability in VLP models by increasing input diversity through local shuffling and sampling.

Findings

LSSA significantly improves transferability across models and tasks.

LSSA outperforms existing advanced attack methods.

LSSA demonstrates robustness on multiple datasets.

Abstract

Visual-Language Pre-training (VLP) models have achieved significant performance across various downstream tasks. However, they remain vulnerable to adversarial examples. While prior efforts focus on improving the adversarial transferability of multimodal adversarial examples through cross-modal interactions, these approaches suffer from overfitting issues, due to a lack of input diversity by relying excessively on information from adversarial examples in one modality when crafting attacks in another. To address this issue, we draw inspiration from strategies in some adversarial training methods and propose a novel attack called Local Shuffle and Sample-based Attack (LSSA). LSSA randomly shuffles one of the local image blocks, thus expanding the original image-text pairs, generating adversarial images, and sampling around them. Then, it utilizes both the original and sampled images to generate the adversarial texts. Extensive experiments on multiple models and datasets demonstrate that LSSA significantly enhances the transferability of multimodal adversarial examples across diverse VLP models and downstream tasks. Moreover, LSSA outperforms other advanced attacks on Large Vision-Language Models.