SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input

TL;DR

SpatialDreamer is a self-supervised framework that synthesizes stereo videos from monocular inputs using a novel diffusion model, addressing data scarcity and ensuring spatio-temporal consistency.

Contribution

It introduces a depth-based video generation module and a self-supervised training framework for stereo video synthesis from monocular videos.

Findings

Outperforms existing methods on benchmark datasets.

Effectively maintains geometric and temporal consistency.

Generates high-quality stereo videos from monocular inputs.

Abstract

Stereo video synthesis from a monocular input is a demanding task in the fields of spatial computing and virtual reality. The main challenges of this task lie on the insufficiency of high-quality paired stereo videos for training and the difficulty of maintaining the spatio-temporal consistency between frames. Existing methods primarily address these issues by directly applying novel view synthesis (NVS) techniques to video, while facing limitations such as the inability to effectively represent dynamic scenes and the requirement for large amounts of training data. In this paper, we introduce a novel self-supervised stereo video synthesis paradigm via a video diffusion model, termed SpatialDreamer, which meets the challenges head-on. Firstly, to address the stereo video data insufficiency, we propose a Depth based Video Generation module DVG, which employs a forward-backward rendering mechanism to generate paired videos with geometric and temporal priors. Leveraging data generated by DVG, we propose RefinerNet along with a self-supervised synthetic framework designed to facilitate efficient and dedicated training. More importantly, we devise a consistency control module, which consists of a metric of stereo deviation strength and a Temporal Interaction Learning module TIL for geometric and temporal consistency ensurance respectively. We evaluated the proposed method against various benchmark methods, with the results showcasing its superior performance.