SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset

TL;DR

SafeSora is a human preference dataset designed to improve safety and alignment in text-to-video generation models by capturing detailed human judgments on helpfulness and harmlessness.

Contribution

The paper introduces the SafeSora dataset with extensive human preference annotations for text-to-video tasks, enabling better alignment of models with human values.

Findings

SafeSora contains 14,711 prompts and 57,333 videos with preference annotations.

The dataset improves training for moderation and alignment of text-to-video models.

Applications demonstrate enhanced safety and alignment in generated videos.

Abstract

To mitigate the risk of harmful outputs from large vision models (LVMs), we introduce the SafeSora dataset to promote research on aligning text-to-video generation with human values. This dataset encompasses human preferences in text-to-video generation tasks along two primary dimensions: helpfulness and harmlessness. To capture in-depth human preferences and facilitate structured reasoning by crowdworkers, we subdivide helpfulness into 4 sub-dimensions and harmlessness into 12 sub-categories, serving as the basis for pilot annotations. The SafeSora dataset includes 14,711 unique prompts, 57,333 unique videos generated by 4 distinct LVMs, and 51,691 pairs of preference annotations labeled by humans. We further demonstrate the utility of the SafeSora dataset through several applications, including training the text-video moderation model and aligning LVMs with human preference by fine-tuning a prompt augmentation module or the diffusion model. These applications highlight its potential as the foundation for text-to-video alignment research, such as human preference modeling and the development and validation of alignment algorithms.