VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videos

TL;DR

VividCam introduces a novel training method enabling diffusion models to learn complex, unconventional camera motions from synthetic videos, overcoming data scarcity and domain shift issues for more artistic video generation.

Contribution

It presents a new training paradigm that uses synthetic data with disentanglement strategies to learn complex camera motions, reducing reliance on real-world videos.

Findings

Synthesizes a wide range of complex camera motions using simple synthetic data.

Effectively mitigates domain shift between synthetic and real videos.

Enables diffusion models to generate more artistic and unconventional videos.

Abstract

Although recent text-to-video generative models are getting more capable of following external camera controls, imposed by either text descriptions or camera trajectories, they still struggle to generalize to unconventional camera motions, which is crucial in creating truly original and artistic videos. The challenge lies in the difficulty of finding sufficient training videos with the intended uncommon camera motions. To address this challenge, we propose VividCam, a training paradigm that enables diffusion models to learn complex camera motions from synthetic videos, releasing the reliance on collecting realistic training videos. VividCam incorporates multiple disentanglement strategies that isolates camera motion learning from synthetic appearance artifacts, ensuring more robust motion representation and mitigating domain shift. We demonstrate that our design synthesizes a wide range of precisely controlled and complex camera motions using surprisingly simple synthetic data. Notably, this synthetic data often consists of basic geometries within a low-poly 3D scene and can be efficiently rendered by engines like Unity. Our video results can be found in https://wuqiuche.github.io/VividCamDemoPage/ .