Mosaic: Multimodal Jailbreak against Closed-Source VLMs via Multi-View Ensemble Optimization

TL;DR

Mosaic is a novel ensemble optimization framework that enhances multimodal jailbreak attacks on closed-source VLMs by reducing surrogate dependency through multi-view and multi-model guidance.

Contribution

It introduces Mosaic, a multi-view ensemble approach that improves attack success on closed-source VLMs by mitigating surrogate dependency and overfitting.

Findings

Achieves state-of-the-art attack success rate.

Demonstrates robustness across heterogeneous surrogate-target settings.

Outperforms existing attack methods in toxicity and success metrics.

Abstract

Vision-Language Models (VLMs) are powerful but remain vulnerable to multimodal jailbreak attacks. Existing attacks mainly rely on either explicit visual prompt attacks or gradient-based adversarial optimization. While the former is easier to detect, the latter produces subtle perturbations that are less perceptible, but is usually optimized and evaluated under homogeneous open-source surrogate-target settings, leaving its effectiveness on commercial closed-source VLMs under heterogeneous settings unclear. To examine this issue, we study different surrogate-target settings and observe a consistent gap between homogeneous and heterogeneous settings, a phenomenon we term surrogate dependency. Motivated by this finding, we propose Mosaic, a Multi-view ensemble optimization framework for multimodal jailbreak against closed-source VLMs, which alleviates surrogate dependency under heterogeneous surrogate-target settings by reducing over-reliance on any single surrogate model and visual view. Specifically, Mosaic incorporates three core components: a Text-Side Transformation module, which perturbs refusal-sensitive lexical patterns; a Multi-View Image Optimization module, which updates perturbations under diverse cropped views to avoid overfitting to a single visual view; and a Surrogate Ensemble Guidance module, which aggregates optimization signals from multiple surrogate VLMs to reduce surrogate-specific bias. Extensive experiments on safety benchmarks demonstrate that Mosaic achieves state-of-the-art Attack Success Rate and Average Toxicity against commercial closed-source VLMs.