Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Models

TL;DR

This paper introduces DO-UAP, a direct optimization-based universal adversarial perturbation method for vision-language models that significantly reduces computational time while maintaining high attack effectiveness.

Contribution

It presents a novel, resource-efficient UAP approach for VLP models, with improved speed and comparable or better attack performance.

Findings

Reduces attack time by 23 times

Maintains high attack success rate

Effective across multiple datasets and models

Abstract

Vision-language pre-training (VLP) models, trained on large-scale image-text pairs, have become widely used across a variety of downstream vision-and-language (V+L) tasks. This widespread adoption raises concerns about their vulnerability to adversarial attacks. Non-universal adversarial attacks, while effective, are often impractical for real-time online applications due to their high computational demands per data instance. Recently, universal adversarial perturbations (UAPs) have been introduced as a solution, but existing generator-based UAP methods are significantly time-consuming. To overcome the limitation, we propose a direct optimization-based UAP approach, termed DO-UAP, which significantly reduces resource consumption while maintaining high attack performance. Specifically, we explore the necessity of multimodal loss design and introduce a useful data augmentation strategy. Extensive experiments conducted on three benchmark VLP datasets, six popular VLP models, and three classical downstream tasks demonstrate the efficiency and effectiveness of DO-UAP. Specifically, our approach drastically decreases the time consumption by 23-fold while achieving a better attack performance.