Image Conductor: Precision Control for Interactive Video Synthesis

TL;DR

Image Conductor enables precise, controllable video synthesis from a single image by separating camera and object motions, using specialized training and guidance techniques to improve control and realism.

Contribution

The paper introduces a novel method for fine-grained control of camera and object motions in video synthesis from a single image, with a new training strategy and guidance technique.

Findings

Demonstrates high precision in motion control

Enables realistic interactive video generation

Outperforms existing methods in control accuracy

Abstract

Filmmaking and animation production often require sophisticated techniques for coordinating camera transitions and object movements, typically involving labor-intensive real-world capturing. Despite advancements in generative AI for video creation, achieving precise control over motion for interactive video asset generation remains challenging. To this end, we propose Image Conductor, a method for precise control of camera transitions and object movements to generate video assets from a single image. An well-cultivated training strategy is proposed to separate distinct camera and object motion by camera LoRA weights and object LoRA weights. To further address cinematographic variations from ill-posed trajectories, we introduce a camera-free guidance technique during inference, enhancing object movements while eliminating camera transitions. Additionally, we develop a trajectory-oriented video motion data curation pipeline for training. Quantitative and qualitative experiments demonstrate our method's precision and fine-grained control in generating motion-controllable videos from images, advancing the practical application of interactive video synthesis. Project webpage available at https://liyaowei-stu.github.io/project/ImageConductor/