Training-free Camera Control for Video Generation

TL;DR

This paper introduces CamTrol, a training-free method for controlling camera movement in video diffusion models, enabling realistic camera motion without finetuning or supervised data, by rearranging noisy latents based on 3D layout modeling.

Contribution

We present CamTrol, a novel plug-and-play approach that achieves camera control in video diffusion models without training or fine-tuning, using layout prior and latent rearrangement.

Findings

Outperforms finetuned methods in video generation and camera motion alignment.

Generalizes across various pretrained video diffusion models.

Enables scalable and complex motion control, including unsupervised 3D video generation.

Abstract

We propose a training-free and robust solution to offer camera movement control for off-the-shelf video diffusion models. Unlike previous work, our method does not require any supervised finetuning on camera-annotated datasets or self-supervised training via data augmentation. Instead, it can be plug-and-play with most pretrained video diffusion models and generate camera-controllable videos with a single image or text prompt as input. The inspiration for our work comes from the layout prior that intermediate latents encode for the generated results, thus rearranging noisy pixels in them will cause the output content to relocate as well. As camera moving could also be seen as a type of pixel rearrangement caused by perspective change, videos can be reorganized following specific camera motion if their noisy latents change accordingly. Building on this, we propose CamTrol, which enables robust camera control for video diffusion models. It is achieved by a two-stage process. First, we model image layout rearrangement through explicit camera movement in 3D point cloud space. Second, we generate videos with camera motion by leveraging the layout prior of noisy latents formed by a series of rearranged images. Extensive experiments have demonstrated its superior performance in both video generation and camera motion alignment compared with other finetuned methods. Furthermore, we show the capability of CamTrol to generalize to various base models, as well as its impressive applications in scalable motion control, dealing with complicated trajectories and unsupervised 3D video generation. Videos available at https://lifedecoder.github.io/CamTrol/.