FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis

TL;DR

FloVD introduces a novel video diffusion model that uses optical flow to enable camera-controllable video synthesis, allowing for natural motion and detailed camera control without requiring ground-truth camera parameters.

Contribution

The paper proposes FloVD, a new approach combining optical flow with diffusion models for camera-controlled video generation, leveraging optical flow for training flexibility and 3D-aware background motion control.

Findings

Outperforms previous methods in camera control accuracy

Synthesizes natural object motion effectively

Supports training with arbitrary videos without ground-truth camera data

Abstract

We present FloVD, a novel video diffusion model for camera-controllable video generation. FloVD leverages optical flow to represent the motions of the camera and moving objects. This approach offers two key benefits. Since optical flow can be directly estimated from videos, our approach allows for the use of arbitrary training videos without ground-truth camera parameters. Moreover, as background optical flow encodes 3D correlation across different viewpoints, our method enables detailed camera control by leveraging the background motion. To synthesize natural object motion while supporting detailed camera control, our framework adopts a two-stage video synthesis pipeline consisting of optical flow generation and flow-conditioned video synthesis. Extensive experiments demonstrate the superiority of our method over previous approaches in terms of accurate camera control and natural object motion synthesis.