VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation

TL;DR

VidCRAFT3 introduces a unified framework for image-to-video generation that allows precise control over camera, object, and lighting, overcoming previous limitations of separate control signals and dataset scarcity.

Contribution

It presents a novel integrated approach with three core components and a new dataset, enabling joint control in image-to-video generation.

Findings

Outperforms existing methods in control accuracy

Achieves higher visual coherence in generated videos

Demonstrates robustness with limited joint annotations

Abstract

Controllable image-to-video (I2V) generation transforms a reference image into a coherent video guided by user-specified control signals. In content creation workflows, precise and simultaneous control over camera motion, object motion, and lighting direction enhances both accuracy and flexibility. However, existing approaches typically treat these control signals separately, largely due to the scarcity of datasets with high-quality joint annotations and mismatched control spaces across modalities. We present VidCRAFT3, a unified and flexible I2V framework that supports both independent and joint control over camera motion, object motion, and lighting direction by integrating three core components. Image2Cloud reconstructs a 3D point cloud from the reference image to enable precise camera motion control. ObjMotionNet encodes sparse object trajectories into multi-scale optical flow features to guide object motion. The Spatial Triple-Attention Transformer integrates lighting direction embeddings via parallel cross-attention. To address the scarcity of jointly annotated data, we curate the VideoLightingDirection (VLD) dataset of synthetic static-scene video clips with per-frame lighting-direction labels, and adopt a three-stage training strategy that enables robust learning without fully joint annotations. Extensive experiments show that VidCRAFT3 outperforms existing methods in control precision and visual coherence. Code and data will be released. Project page: https://sixiaozheng.github.io/VidCRAFT3/.