SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints
Jianhong Bai, Menghan Xia, Xintao Wang, Ziyang Yuan, Xiao Fu, Zuozhu, Liu, Haoji Hu, Pengfei Wan, Di Zhang

TL;DR
SynCamMaster enhances pre-trained text-to-video models with a multi-view synchronization module, enabling consistent, open-world multi-camera video generation from arbitrary viewpoints, supported by a hybrid training scheme and a new dataset.
Contribution
It introduces a plug-and-play multi-view synchronization module and a hybrid training scheme for multi-camera video generation from diverse viewpoints.
Findings
Achieves dynamic consistency across viewpoints in generated videos.
Enables re-rendering of videos from novel viewpoints.
Provides a new dataset for multi-view video synthesis.
Abstract
Recent advancements in video diffusion models have shown exceptional abilities in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dynamic consistency across various viewpoints, a highly desirable feature for applications such as virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsAdvanced Vision and Imaging · Video Coding and Compression Technologies · Computer Graphics and Visualization Techniques
MethodsDiffusion
