Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models

TL;DR

This paper analyzes adversarial transferability in Multimodal Large Language Models, identifies key influencing factors, and proposes data augmentation methods to enhance transferability, with implications for both harmful and protective societal applications.

Contribution

It provides a detailed analysis of adversarial transferability among MLLMs, identifies key factors affecting transferability, and introduces two semantic-level data augmentation techniques to improve it.

Findings

Transferability exists in cross-LLM scenarios with the same vision encoder.

Two key factors influence adversarial transferability.

Proposed methods boost transferability across MLLMs.

Abstract

Multimodal Large Language Models (MLLMs) demonstrate exceptional performance in cross-modality interaction, yet they also suffer adversarial vulnerabilities. In particular, the transferability of adversarial examples remains an ongoing challenge. In this paper, we specifically analyze the manifestation of adversarial transferability among MLLMs and identify the key factors that influence this characteristic. We discover that the transferability of MLLMs exists in cross-LLM scenarios with the same vision encoder and indicate \underline{\textit{two key Factors}} that may influence transferability. We provide two semantic-level data augmentation methods, Adding Image Patch (AIP) and Typography Augment Transferability Method (TATM), which boost the transferability of adversarial examples across MLLMs. To explore the potential impact in the real world, we utilize two tasks that can have both negative and positive societal impacts: \ding{182} Harmful Content Insertion and \ding{183} Information Protection.