Can video generation replace cinematographers? Research on the cinematic language of generated video

TL;DR

This paper advances text-to-video generation by integrating cinematic language control through a new dataset, specialized models, and evaluation methods, aiming to produce videos with professional cinematic quality.

Contribution

It introduces a cinematic language dataset, CameraDiff for control, and CameraCLIP with CLIPLoRA for evaluation and style blending, enhancing cinematic coherence in generated videos.

Findings

CameraDiff provides stable cinematic control.

CameraCLIP achieves high cinematic alignment accuracy.

CLIPLoRA improves multi-shot style blending.

Abstract

Recent advancements in text-to-video (T2V) generation have leveraged diffusion models to enhance visual coherence in videos synthesized from textual descriptions. However, existing research primarily focuses on object motion, often overlooking cinematic language, which is crucial for conveying emotion and narrative pacing in cinematography. To address this, we propose a threefold approach to improve cinematic control in T2V models. First, we introduce a meticulously annotated cinematic language dataset with twenty subcategories, covering shot framing, shot angles, and camera movements, enabling models to learn diverse cinematic styles. Second, we present CameraDiff, which employs LoRA for precise and stable cinematic control, ensuring flexible shot generation. Third, we propose CameraCLIP, designed to evaluate cinematic alignment and guide multi-shot composition. Building on CameraCLIP, we introduce CLIPLoRA, a CLIP-guided dynamic LoRA composition method that adaptively fuses multiple pre-trained cinematic LoRAs, enabling smooth transitions and seamless style blending. Experimental results demonstrate that CameraDiff ensures stable and precise cinematic control, CameraCLIP achieves an R@1 score of 0.83, and CLIPLoRA significantly enhances multi-shot composition within a single video, bridging the gap between automated video generation and professional cinematography.\textsuperscript{1}