AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers
Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin,, Willi Menapace, Andrea Tagliasacchi, David B. Lindell, Sergey Tulyakov

TL;DR
This paper introduces AC3D, a novel approach for precise 3D camera control in text-to-video models, improving quality and training efficiency by analyzing camera motion, optimizing conditioning, and curating a new dataset.
Contribution
We analyze camera motion from first principles, optimize conditioning strategies, and curate a new dataset, leading to the state-of-the-art AC3D model for camera-controlled video generation.
Findings
Camera motion is low-frequency, enabling better conditioning strategies.
Limiting camera conditioning to certain layers reduces parameters and improves quality.
A new dataset of 20K dynamic videos enhances model performance.
Abstract
Numerous works have recently integrated 3D camera control into foundational text-to-video models, but the resulting camera control is often imprecise, and video generation quality suffers. In this work, we analyze camera motion from a first principles perspective, uncovering insights that enable precise 3D camera manipulation without compromising synthesis quality. First, we determine that motion induced by camera movements in videos is low-frequency in nature. This motivates us to adjust train and test pose conditioning schedules, accelerating training convergence while improving visual and motion quality. Then, by probing the representations of an unconditional video diffusion transformer, we observe that they implicitly perform camera pose estimation under the hood, and only a sub-portion of their layers contain the camera information. This suggested us to limit the injection of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsIndustrial Vision Systems and Defect Detection · CCD and CMOS Imaging Sensors
MethodsSPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings · Diffusion
