JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering

TL;DR

JPS introduces a novel method combining visual perturbations and textual steering to effectively jailbreak multimodal large language models, significantly improving attack success and malicious intent fulfillment rates.

Contribution

The paper proposes JPS, a new collaborative visual and textual attack framework that enhances jailbreak success and malicious response quality in multimodal large language models.

Findings

JPS achieves state-of-the-art attack success rates across various benchmarks.

The Malicious Intent Fulfillment Rate (MIFR) metric effectively measures attack quality.

Iterative co-optimization of visual and textual components improves attack performance.

Abstract

Jailbreak attacks against multimodal large language Models (MLLMs) are a significant research focus. Current research predominantly focuses on maximizing attack success rate (ASR), often overlooking whether the generated responses actually fulfill the attacker's malicious intent. This oversight frequently leads to low-quality outputs that bypass safety filters but lack substantial harmful content. To address this gap, we propose JPS, \underline{J}ailbreak MLLMs with collaborative visual \underline{P}erturbation and textual \underline{S}teering, which achieves jailbreaks via corporation of visual image and textually steering prompt. Specifically, JPS utilizes target-guided adversarial image perturbations for effective safety bypass, complemented by "steering prompt" optimized via a multi-agent system to specifically guide LLM responses fulfilling the attackers' intent. These visual and textual components undergo iterative co-optimization for enhanced performance. To evaluate the quality of attack outcomes, we propose the Malicious Intent Fulfillment Rate (MIFR) metric, assessed using a Reasoning-LLM-based evaluator. Our experiments show JPS sets a new state-of-the-art in both ASR and MIFR across various MLLMs and benchmarks, with analyses confirming its efficacy. Codes are available at \href{https://github.com/thu-coai/JPS}{https://github.com/thu-coai/JPS}. \color{warningcolor}{Warning: This paper contains potentially sensitive contents.}