Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models

TL;DR

This paper introduces MAPA, a multi-turn adaptive prompting attack that effectively increases the maliciousness of responses in large vision-language models by iteratively refining attack strategies, surpassing existing methods in success rate.

Contribution

The paper proposes MAPA, a novel multi-turn adaptive attack method that enhances attack success rates on LVLMs by dynamically adjusting prompts across turns.

Findings

MAPA improves attack success rates by 11-35% over state-of-the-art methods.

MAPA effectively bypasses safety defenses in LVLMs.

Iterative refinement enhances malicious response elicitation.

Abstract

Multi-turn jailbreak attacks are effective against text-only large language models (LLMs) by gradually introducing malicious content across turns. When extended to large vision-language models (LVLMs), we find that naively adding visual inputs can cause existing multi-turn jailbreaks to be easily defended. For example, overly malicious visual input will easily trigger the defense mechanism of safety-aligned LVLMs, making the response more conservative. To address this, we propose MAPA: a multi-turn adaptive prompting attack that 1) at each turn, alternates text-vision attack actions to elicit the most malicious response; and 2) across turns, adjusts the attack trajectory through iterative back-and-forth refinement to gradually amplify response maliciousness. This two-level design enables MAPA to consistently outperform state-of-the-art methods, improving attack success rates by 11-35% on recent benchmarks against LLaVA-V1.6-Mistral-7B, Qwen2.5-VL-7B-Instruct, Llama-3.2-Vision-11B-Instruct and GPT-4o-mini.