SpatialMe: Stereo Video Conversion Using Depth-Warping and Blend-Inpainting

TL;DR

SpatialMe is a novel framework for converting monocular videos into high-quality stereo videos using depth-warping, blend-inpainting, and a new dataset, addressing key challenges in fidelity and data scarcity.

Contribution

The paper introduces SpatialMe, a stereo video conversion method with a mask-based hierarchy feature update and disparity expansion, plus a new real-world stereo video dataset StereoV1K.

Findings

Outperforms state-of-the-art in stereo video quality

Effectively addresses foreground bleeding issues

Provides a new high-quality stereo video dataset

Abstract

Stereo video conversion aims to transform monocular videos into immersive stereo format. Despite the advancements in novel view synthesis, it still remains two major challenges: i) difficulty of achieving high-fidelity and stable results, and ii) insufficiency of high-quality stereo video data. In this paper, we introduce SpatialMe, a novel stereo video conversion framework based on depth-warping and blend-inpainting. Specifically, we propose a mask-based hierarchy feature update (MHFU) refiner, which integrate and refine the outputs from designed multi-branch inpainting module, using feature update unit (FUU) and mask mechanism. We also propose a disparity expansion strategy to address the problem of foreground bleeding. Furthermore, we conduct a high-quality real-world stereo video dataset -- StereoV1K, to alleviate the data shortage. It contains 1000 stereo videos captured in real-world at a resolution of 1180 x 1180, covering various indoor and outdoor scenes. Extensive experiments demonstrate the superiority of our approach in generating stereo videos over state-of-the-art methods.