MSR-Align: Policy-Grounded Multimodal Alignment for Safety-Aware Reasoning in Vision-Language Models

TL;DR

This paper introduces MSR-Align, a new multimodal safety reasoning dataset designed to improve safety alignment in vision-language models by enabling fine-grained, policy-grounded reasoning across modalities.

Contribution

The paper presents MSR-Align, a novel dataset for safety reasoning in VLMs, and demonstrates its effectiveness in enhancing model robustness against multimodal safety threats.

Findings

Fine-tuning on MSR-Align improves robustness against jailbreak attacks.

MSR-Align enhances safety alignment without sacrificing reasoning performance.

The dataset supports scalable, policy-grounded multimodal safety reasoning.

Abstract

Vision-Language Models (VLMs) have achieved remarkable progress in multimodal reasoning tasks through enhanced chain-of-thought capabilities. However, this advancement also introduces novel safety risks, as these models become increasingly vulnerable to harmful multimodal prompts that can trigger unethical or unsafe behaviors. Existing safety alignment approaches, primarily designed for unimodal language models, fall short in addressing the complex and nuanced threats posed by multimodal inputs. Moreover, current safety datasets lack the fine-grained, policy-grounded reasoning required to robustly align reasoning-capable VLMs. In this work, we introduce {MSR-Align}, a high-quality Multimodal Safety Reasoning dataset tailored to bridge this gap. MSR-Align supports fine-grained, deliberative reasoning over standardized safety policies across both vision and text modalities. Our data generation pipeline emphasizes multimodal diversity, policy-grounded reasoning, and rigorous quality filtering using strong multimodal judges. Extensive experiments demonstrate that fine-tuning VLMs on MSR-Align substantially improves robustness against both textual and vision-language jailbreak attacks, while preserving or enhancing general reasoning performance. MSR-Align provides a scalable and effective foundation for advancing the safety alignment of reasoning-capable VLMs. Our dataset is made publicly available at https://huggingface.co/datasets/Leigest/MSR-Align.