ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models

TL;DR

ShotAdapter introduces a novel framework for generating multi-shot videos from text prompts, enabling discrete shot transitions, character consistency, and user control over shot content and timing using diffusion models.

Contribution

The paper presents a new dataset collection pipeline and architectural extensions for diffusion models to enable text-to-multi-shot video generation with shot-specific control.

Findings

Fine-tuning a pre-trained model enables multi-shot video generation.

The approach outperforms existing baselines.

The method ensures character and background consistency across shots.

Abstract

Current diffusion-based text-to-video methods are limited to producing short video clips of a single shot and lack the capability to generate multi-shot videos with discrete transitions where the same character performs distinct activities across the same or different backgrounds. To address this limitation we propose a framework that includes a dataset collection pipeline and architectural extensions to video diffusion models to enable text-to-multi-shot video generation. Our approach enables generation of multi-shot videos as a single video with full attention across all frames of all shots, ensuring character and background consistency, and allows users to control the number, duration, and content of shots through shot-specific conditioning. This is achieved by incorporating a transition token into the text-to-video model to control at which frames a new shot begins and a local attention masking strategy which controls the transition token's effect and allows shot-specific prompting. To obtain training data we propose a novel data collection pipeline to construct a multi-shot video dataset from existing single-shot video datasets. Extensive experiments demonstrate that fine-tuning a pre-trained text-to-video model for a few thousand iterations is enough for the model to subsequently be able to generate multi-shot videos with shot-specific control, outperforming the baselines. You can find more details in https://shotadapter.github.io/