Towards Adversarial Attack on Vision-Language Pre-training Models

TL;DR

This paper investigates the adversarial robustness of vision-language pre-training models, analyzes attack strategies, and introduces a novel collaborative multimodal attack method to improve understanding and robustness of such models.

Contribution

It presents the first comprehensive analysis of adversarial attacks on VLP models and proposes Co-Attack, a new multimodal attack technique that enhances attack effectiveness.

Findings

Co-Attack outperforms existing attack methods on V+L tasks

Key factors influencing attack success identified

Insights into robustness guide future defenses

Abstract

While vision-language pre-training model (VLP) has shown revolutionary improvements on various vision-language (V+L) tasks, the studies regarding its adversarial robustness remain largely unexplored. This paper studied the adversarial attack on popular VLP models and V+L tasks. First, we analyzed the performance of adversarial attacks under different settings. By examining the influence of different perturbed objects and attack targets, we concluded some key observations as guidance on both designing strong multimodal adversarial attack and constructing robust VLP models. Second, we proposed a novel multimodal attack method on the VLP models called Collaborative Multimodal Adversarial Attack (Co-Attack), which collectively carries out the attacks on the image modality and the text modality. Experimental results demonstrated that the proposed method achieves improved attack performances on different V+L downstream tasks and VLP models. The analysis observations and novel attack method hopefully provide new understanding into the adversarial robustness of VLP models, so as to contribute their safe and reliable deployment in more real-world scenarios. Code is available at https://github.com/adversarial-for-goodness/Co-Attack.