Enhancing Scene Transition Awareness in Video Generation via Post-Training

TL;DR

This paper introduces a post-training method on a new dataset to improve scene transition detection in text-to-video models, enabling longer, multi-scene video generation with better coherence.

Contribution

The paper presents the TAV dataset and demonstrates that post-training on it enhances scene transition awareness in video generation models.

Findings

Improved scene transition detection accuracy.

Enhanced multi-scene video coherence.

Maintained image quality during longer videos.

Abstract

Recent advances in AI-generated video have shown strong performance on \emph{text-to-video} tasks, particularly for short clips depicting a single scene. However, current models struggle to generate longer videos with coherent scene transitions, primarily because they cannot infer when a transition is needed from the prompt. Most open-source models are trained on datasets consisting of single-scene video clips, which limits their capacity to learn and respond to prompts requiring multiple scenes. Developing scene transition awareness is essential for multi-scene generation, as it allows models to identify and segment videos into distinct clips by accurately detecting transitions. To address this, we propose the \textbf{Transition-Aware Video} (TAV) dataset, which consists of preprocessed video clips with multiple scene transitions. Our experiment shows that post-training on the \textbf{TAV} dataset improves prompt-based scene transition understanding, narrows the gap between required and generated scenes, and maintains image quality.