PRISM: Robust VLM Alignment with Principled Reasoning for Integrated Safety in Multimodality

TL;DR

PRISM is a structured reasoning framework that enhances the safety and robustness of vision-language models against complex multimodal threats while maintaining utility.

Contribution

It introduces a four-stage reasoning process combined with Monte Carlo Tree Search for improved multimodal safety in VLMs, outperforming existing methods.

Findings

Reduces attack success rates on JailbreakV-28K and VLBreak datasets.

Improves robustness against adaptive multimodal attacks.

Maintains utility on benign multimodal benchmarks.

Abstract

Safeguarding vision-language models (VLMs) is a critical challenge, as existing methods often suffer from over-defense, which harms utility, or rely on shallow alignment, failing to detect complex threats that require deep reasoning. To this end, we introduc PRISM (Principled Reasoning for Integrated Safety in Multimodality), a System 2-like framework that aligns VLMs through a structured four-stage reasoning process explicitly designed to handle three distinct categories of multimodal safety violations. Our framework consists of two key components: a structured reasoning pipeline that analyzes each violation category in dedicated stages, and PRISM-DPO, generated via Monte Carlo Tree Search (MCTS) to refine reasoning quality through Direct Preference Optimization. Comprehensive evaluations show that PRISM substantially reduces attack success rates on JailbreakV-28K and VLBreak, improves robustness against adaptive attacks, and generalizes to out-of-distribution multi-image threats, while better preserving model utility on benign multimodal benchmarks. Our code, data, and model weights available at https://github.com/SaFoLab-WISC/PRISM.