ImmersePro: End-to-End Stereo Video Synthesis Via Implicit Disparity Learning

TL;DR

ImmersePro is a novel end-to-end framework that converts single-view videos into stereo videos using implicit disparity learning, leveraging a new dataset and attention mechanisms for high-quality stereo synthesis.

Contribution

The paper introduces ImmersePro, a dual-branch architecture with implicit disparity guidance and a large-scale stereo video dataset for improved stereo video generation.

Findings

Significant quantitative improvements over existing methods.

Effective stereo video synthesis from monocular videos.

Introduction of the large-scale YouTube-SBS dataset.

Abstract

We introduce \textit{ImmersePro}, an innovative framework specifically designed to transform single-view videos into stereo videos. This framework utilizes a novel dual-branch architecture comprising a disparity branch and a context branch on video data by leveraging spatial-temporal attention mechanisms. \textit{ImmersePro} employs implicit disparity guidance, enabling the generation of stereo pairs from video sequences without the need for explicit disparity maps, thus reducing potential errors associated with disparity estimation models. In addition to the technical advancements, we introduce the YouTube-SBS dataset, a comprehensive collection of 423 stereo videos sourced from YouTube. This dataset is unprecedented in its scale, featuring over 7 million stereo pairs, and is designed to facilitate training and benchmarking of stereo video generation models. Our experiments demonstrate the effectiveness of \textit{ImmersePro} in producing high-quality stereo videos, offering significant improvements over existing methods. Compared to the best competitor stereo-from-mono we quantitatively improve the results by 11.76\% (L1), 6.39\% (SSIM), and 5.10\% (PSNR).