Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models

TL;DR

This paper introduces Hierarchical Refinement Attack (HRA), a novel universal multimodal attack framework on vision-language models that improves transferability and efficiency by hierarchically refining perturbations in both image and text modalities.

Contribution

The paper presents a new hierarchical approach to universal multimodal attacks on VLP models, reducing computational costs and enhancing transferability across tasks and datasets.

Findings

HRA achieves superior transferability compared to existing methods.

HRA effectively refines perturbations in both image and text modalities.

Extensive experiments validate the robustness of the proposed attack.

Abstract

Existing adversarial attacks for VLP models are mostly sample-specific, resulting in substantial computational overhead when scaled to large datasets or new scenarios. To overcome this limitation, we propose Hierarchical Refinement Attack (HRA), a multimodal universal attack framework for VLP models. For the image modality, we refine the optimization path by leveraging a temporal hierarchy of historical and estimated future gradients to avoid local minima and stabilize universal perturbation learning. For the text modality, it hierarchically models textual importance by considering both intra- and inter-sentence contributions to identify globally influential words, which are then used as universal text perturbations. Extensive experiments across various downstream tasks, VLP models, and datasets, demonstrate the superior transferability of the proposed universal multimodal attacks.