OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models

TL;DR

OutSafe-Bench is a comprehensive multimodal safety evaluation benchmark for large language models, featuring a large annotated dataset across four modalities and introducing new metrics and evaluation frameworks.

Contribution

It presents the first extensive multimodal safety benchmark with a large dataset, novel risk scoring, and a fair evaluation framework for assessing MLLMs.

Findings

Nine state-of-the-art MLLMs show significant safety vulnerabilities.

The benchmark covers four modalities with over 18,000 prompts and thousands of multimedia samples.

The proposed metrics and frameworks improve the robustness and fairness of safety evaluations.

Abstract

Since Multimodal Large Language Models (MLLMs) are increasingly being integrated into everyday tools and intelligent agents, growing concerns have arisen regarding their possible output of unsafe contents, ranging from toxic language and biased imagery to privacy violations and harmful misinformation. Current safety benchmarks remain highly limited in both modality coverage and performance evaluations, often neglecting the extensive landscape of content safety. In this work, we introduce OutSafe-Bench, the first most comprehensive content safety evaluation test suite designed for the multimodal era. OutSafe-Bench includes a large-scale dataset that spans four modalities, featuring over 18,000 bilingual (Chinese and English) text prompts, 4,500 images, 450 audio clips and 450 videos, all systematically annotated across nine critical content risk categories. In addition to the dataset, we introduce a Multidimensional Cross Risk Score (MCRS), a novel metric designed to model and assess overlapping and correlated content risks across different categories. To ensure fair and robust evaluation, we propose FairScore, an explainable automated multi-reviewer weighted aggregation framework. FairScore selects top-performing models as adaptive juries, thereby mitigating biases from single-model judgments and enhancing overall evaluation reliability. Our evaluation of nine state-of-the-art MLLMs reveals persistent and substantial safety vulnerabilities, underscoring the pressing need for robust safeguards in MLLMs.