CamI2V: Camera-Controlled Image-to-Video Diffusion Model

TL;DR

This paper introduces CamI2V, a novel camera-controlled image-to-video diffusion model that improves geometry consistency and controllability by using epipolar attention and robust evaluation, achieving significant performance gains.

Contribution

It proposes a new method for modeling noisy cross-frame interactions with epipolar attention and robust evaluation, enhancing camera controllability in video diffusion models.

Findings

25.64% improvement in camera controllability on RealEstate10K

Robust performance in dynamic and occluded scenarios

Efficient training and inference with limited memory

Abstract

Recent advancements have integrated camera pose as a user-friendly and physics-informed condition in video diffusion models, enabling precise camera control. In this paper, we identify one of the key challenges as effectively modeling noisy cross-frame interactions to enhance geometry consistency and camera controllability. We innovatively associate the quality of a condition with its ability to reduce uncertainty and interpret noisy cross-frame features as a form of noisy condition. Recognizing that noisy conditions provide deterministic information while also introducing randomness and potential misguidance due to added noise, we propose applying epipolar attention to only aggregate features along corresponding epipolar lines, thereby accessing an optimal amount of noisy conditions. Additionally, we address scenarios where epipolar lines disappear, commonly caused by rapid camera movements, dynamic objects, or occlusions, ensuring robust performance in diverse environments. Furthermore, we develop a more robust and reproducible evaluation pipeline to address the inaccuracies and instabilities of existing camera control metrics. Our method achieves a 25.64% improvement in camera controllability on the RealEstate10K dataset without compromising dynamics or generation quality and demonstrates strong generalization to out-of-domain images. Training and inference require only 24GB and 12GB of memory, respectively, for 16-frame sequences at 256x256 resolution. We will release all checkpoints, along with training and evaluation code. Dynamic videos are best viewed at https://zgctroy.github.io/CamI2V.