LAMP: Learning Universal Adversarial Perturbations for Multi-Image Tasks via Pre-trained Models

TL;DR

LAMP is a novel black-box attack method that learns universal adversarial perturbations to effectively compromise multi-image vision-language models by exploiting attention mechanisms and cross-image influences.

Contribution

The paper introduces LAMP, a new approach for generating universal adversarial perturbations targeting multi-image models, with novel constraints and loss functions for improved attack success.

Findings

LAMP outperforms state-of-the-art baselines in attack success rates.

LAMP effectively disrupts multi-image vision-language tasks.

LAMP demonstrates robustness across various models and tasks.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable performance across vision-language tasks. Recent advancements allow these models to process multiple images as inputs. However, the vulnerabilities of multi-image MLLMs remain unexplored. Existing adversarial attacks focus on single-image settings and often assume a white-box threat model, which is impractical in many real-world scenarios. This paper introduces LAMP, a black-box method for learning Universal Adversarial Perturbations (UAPs) targeting multi-image MLLMs. LAMP applies an attention-based constraint that prevents the model from effectively aggregating information across images. LAMP also introduces a novel cross-image contagious constraint that forces perturbed tokens to influence clean tokens, spreading adversarial effects without requiring all inputs to be modified. Additionally, an index-attention suppression loss enables a robust position-invariant attack. Experimental results show that LAMP outperforms SOTA baselines and achieves the highest attack success rates across multiple vision-language tasks and models.