T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models

TL;DR

This paper introduces T2VSafetyBench, a comprehensive benchmark for evaluating safety aspects of text-to-video models, addressing a critical gap in understanding and mitigating risks like illegal or unethical content generation.

Contribution

The paper presents a new safety benchmark with 12 aspects, a malicious prompt dataset, and evaluations revealing diverse model strengths and safety trade-offs.

Findings

No single model excels in all safety aspects

High correlation between GPT-4 and manual safety assessments

Trade-off observed between usability and safety

Abstract

The recent development of Sora leads to a new era in text-to-video (T2V) generation. Along with this comes the rising concern about its security risks. The generated videos may contain illegal or unethical content, and there is a lack of comprehensive quantitative understanding of their safety, posing a challenge to their reliability and practical deployment. Previous evaluations primarily focus on the quality of video generation. While some evaluations of text-to-image models have considered safety, they cover fewer aspects and do not address the unique temporal risk inherent in video generation. To bridge this research gap, we introduce T2VSafetyBench, a new benchmark designed for conducting safety-critical assessments of text-to-video models. We define 12 critical aspects of video generation safety and construct a malicious prompt dataset including real-world prompts, LLM-generated prompts and jailbreak attack-based prompts. Based on our evaluation results, we draw several important findings, including: 1) no single model excels in all aspects, with different models showing various strengths; 2) the correlation between GPT-4 assessments and manual reviews is generally high; 3) there is a trade-off between the usability and safety of text-to-video generative models. This indicates that as the field of video generation rapidly advances, safety risks are set to surge, highlighting the urgency of prioritizing video safety. We hope that T2VSafetyBench can provide insights for better understanding the safety of video generation in the era of generative AI.