DissolveStereo: Coarse Depth Injection for Zero-Shot Stereo Video Generation

TL;DR

DissolveStereo is a zero-shot framework that generates high-quality stereo videos with consistent depth and temporal coherence using diffusion priors and innovative latent space refinement.

Contribution

It introduces a novel zero-shot stereo video generation method leveraging diffusion models, with a noisy restart strategy and dissolved depth maps for improved coherence.

Findings

Achieves 11.7% improvement in epipolar consistency (MEt3R score)

User studies show 8.0% higher perceived frame quality

User studies show 10.9% higher perceived temporal coherence

Abstract

Generating high-quality stereo videos requires consistent depth perception and temporal coherence across frames. Despite advances in image and video synthesis using diffusion models, producing high-quality stereo videos remains a challenging task due to the difficulty of maintaining consistent temporal and spatial coherence between left and right views. We introduce DissolveStereo, a novel framework for zero-shot stereo video generation that leverages video diffusion priors without requiring paired training data. Our key innovations include a noisy restart strategy to initialize stereo-aware latent representations and an iterative refinement process that progressively harmonizes the latent space, addressing issues like temporal flickering and view inconsistencies. Importantly, we propose the use of dissolved depth maps to streamline latent space operations by reducing high-frequency depth information. Our comprehensive evaluations, including quantitative metrics and user studies, demonstrate that DissolveStereo produces high-quality stereo videos with enhanced depth consistency and temporal smoothness. In terms of epipolar consistency, our method achieves an 11.7% improvement in MEt3R score over the current state-of-the-art. Furthermore, user studies indicate strong perceptual gains over the previous arts, with an 8.0% higher perceived frame quality and 10.9% higher perceived temporal coherence. Our code is in https://github.com/shijianjian/DissolveStereo.