ShotVerse: Advancing Cinematic Camera Control for Text-Driven Multi-Shot Video Creation

TL;DR

ShotVerse introduces a novel framework that combines vision-language models and automated calibration to enable precise, cinematic multi-shot video creation driven by textual prompts, overcoming manual and implicit control limitations.

Contribution

The paper presents a data-centric approach with a new planning-control framework and a curated cinematic dataset for text-driven multi-shot video generation.

Findings

Achieves camera-accurate, consistent multi-shot videos from text

Bridges gap between textual prompts and manual plotting

Outperforms existing methods in cinematic quality

Abstract

Text-driven video generation has democratized film creation, but camera control in cinematic multi-shot scenarios remains a significant block. Implicit textual prompts lack precision, while explicit trajectory conditioning imposes prohibitive manual overhead and often triggers execution failures in current models. To overcome this bottleneck, we propose a data-centric paradigm shift, positing that aligned (Caption, Trajectory, Video) triplets form an inherent joint distribution that can connect automated plotting and precise execution. Guided by this insight, we present ShotVerse, a "Plan-then-Control" framework that decouples generation into two collaborative agents: a VLM (Vision-Language Model)-based Planner that leverages spatial priors to obtain cinematic, globally aligned trajectories from text, and a Controller that renders these trajectories into multi-shot video content via a camera adapter. Central to our approach is the construction of a data foundation: we design an automated multi-shot camera calibration pipeline aligns disjoint single-shot trajectories into a unified global coordinate system. This facilitates the curation of ShotVerse-Bench, a high-fidelity cinematic dataset with a three-track evaluation protocol that serves as the bedrock for our framework. Extensive experiments demonstrate that ShotVerse effectively bridges the gap between unreliable textual control and labor-intensive manual plotting, achieving superior cinematic aesthetics and generating multi-shot videos that are both camera-accurate and cross-shot consistent.