SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints

TL;DR

SynCamMaster enhances pre-trained text-to-video models with a multi-view synchronization module, enabling consistent, open-world multi-camera video generation from arbitrary viewpoints, supported by a hybrid training scheme and a new dataset.

Contribution

It introduces a plug-and-play multi-view synchronization module and a hybrid training scheme for multi-camera video generation from diverse viewpoints.

Findings

Achieves dynamic consistency across viewpoints in generated videos.

Enables re-rendering of videos from novel viewpoints.

Provides a new dataset for multi-view video synthesis.

Abstract

Recent advancements in video diffusion models have shown exceptional abilities in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dynamic consistency across various viewpoints, a highly desirable feature for applications such as virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we design a hybrid training scheme that leverages multi-camera images and monocular videos to supplement Unreal Engine-rendered multi-camera videos. Furthermore, our method enables intriguing extensions, such as re-rendering a video from novel viewpoints. We also release a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project page: https://jianhongbai.github.io/SynCamMaster/.