StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos

TL;DR

This paper introduces StereoCrafter, a diffusion-based framework that converts monocular videos into high-fidelity stereoscopic 3D content, enhancing immersive experiences for 3D displays and devices.

Contribution

It presents a novel combination of depth-based video warping, occlusion handling, and inpainting using foundation models, with strategies for varying input lengths and high-quality dataset creation.

Findings

Significant improvement in 2D-to-3D video conversion quality

Effective use of foundation models for stereoscopic inpainting

Practical framework for immersive 3D content creation

Abstract

This paper presents a novel framework for converting 2D videos to immersive stereoscopic 3D, addressing the growing demand for 3D content in immersive experience. Leveraging foundation models as priors, our approach overcomes the limitations of traditional methods and boosts the performance to ensure the high-fidelity generation required by the display devices. The proposed system consists of two main steps: depth-based video splatting for warping and extracting occlusion mask, and stereo video inpainting. We utilize pre-trained stable video diffusion as the backbone and introduce a fine-tuning protocol for the stereo video inpainting task. To handle input video with varying lengths and resolutions, we explore auto-regressive strategies and tiled processing. Finally, a sophisticated data processing pipeline has been developed to reconstruct a large-scale and high-quality dataset to support our training. Our framework demonstrates significant improvements in 2D-to-3D video conversion, offering a practical solution for creating immersive content for 3D devices like Apple Vision Pro and 3D displays. In summary, this work contributes to the field by presenting an effective method for generating high-quality stereoscopic videos from monocular input, potentially transforming how we experience digital media.