CI-VID: A Coherent Interleaved Text-Video Dataset

TL;DR

CI-VID is a large, coherent multi-clip video dataset with text captions, enabling improved training for models to generate story-like, temporally consistent video sequences, advancing text-to-video generation research.

Contribution

The paper introduces CI-VID, a novel dataset supporting coherent multi-clip video generation with detailed captions, and establishes a comprehensive benchmark for evaluating such models.

Findings

Models trained on CI-VID show improved accuracy in video sequence generation.

CI-VID enables better temporal coherence and transition modeling in generated videos.

The dataset facilitates story-driven video creation with smooth visual transitions.

Abstract

Text-to-video (T2V) generation has recently attracted considerable attention, resulting in the development of numerous high-quality datasets that have propelled progress in this area. However, existing public datasets are primarily composed of isolated text-video (T-V) pairs and thus fail to support the modeling of coherent multi-clip video sequences. To address this limitation, we introduce CI-VID, a dataset that moves beyond isolated text-to-video (T2V) generation toward text-and-video-to-video (TV2V) generation, enabling models to produce coherent, multi-scene video sequences. CI-VID contains over 340,000 samples, each featuring a coherent sequence of video clips with text captions that capture both the individual content of each clip and the transitions between them, enabling visually and textually grounded generation. To further validate the effectiveness of CI-VID, we design a comprehensive, multi-dimensional benchmark incorporating human evaluation, VLM-based assessment, and similarity-based metrics. Experimental results demonstrate that models trained on CI-VID exhibit significant improvements in both accuracy and content consistency when generating video sequences. This facilitates the creation of story-driven content with smooth visual transitions and strong temporal coherence, underscoring the quality and practical utility of the CI-VID dataset We release the CI-VID dataset and the accompanying code for data construction and evaluation at: https://github.com/ymju-BAAI/CI-VID