PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking

TL;DR

This paper introduces a novel jailbreak method for large vision-language models that exploits their reasoning process by decomposing harmful instructions into benign visual components, revealing a significant vulnerability.

Contribution

The authors propose a ROP-inspired framework that effectively bypasses safety measures by leveraging the compositional reasoning of LVLMs, demonstrating its success on multiple benchmarks.

Findings

Achieves over 0.90 attack success rate on SafeBench

Outperforms existing baselines significantly

Reveals a critical vulnerability in LVLM safety defenses

Abstract

The increasing sophistication of large vision-language models (LVLMs) has been accompanied by advances in safety alignment mechanisms designed to prevent harmful content generation. However, these defenses remain vulnerable to sophisticated adversarial attacks. Existing jailbreak methods typically rely on direct and semantically explicit prompts, overlooking subtle vulnerabilities in how LVLMs compose information over multiple reasoning steps. In this paper, we propose a novel and effective jailbreak framework inspired by Return-Oriented Programming (ROP) techniques from software security. Our approach decomposes a harmful instruction into a sequence of individually benign visual gadgets. A carefully engineered textual prompt directs the sequence of inputs, prompting the model to integrate the benign visual gadgets through its reasoning process to produce a coherent and harmful output. This makes the malicious intent emergent and difficult to detect from any single component. We validate our method through extensive experiments on established benchmarks including SafeBench and MM-SafetyBench, targeting popular LVLMs. Results show that our approach consistently and substantially outperforms existing baselines on state-of-the-art models, achieving near-perfect attack success rates (over 0.90 on SafeBench) and improving ASR by up to 0.39. Our findings reveal a critical and underexplored vulnerability that exploits the compositional reasoning abilities of LVLMs, highlighting the urgent need for defenses that secure the entire reasoning process.