DreamStereo: Towards Real-Time Stereo Inpainting for HD Videos

TL;DR

This paper introduces a real-time stereo video inpainting method that leverages novel warping and projection techniques to efficiently fill occluded regions, achieving high-quality results at 25 FPS on HD videos.

Contribution

The authors propose three interconnected components—GAPW, PBDP, and SASI—that improve stereo inpainting accuracy and efficiency, enabling real-time HD video processing.

Findings

Achieves over 70% reduction in redundant computation tokens.

Enables real-time HD stereo video inpainting at 25 FPS on a single GPU.

Produces visually coherent and temporally consistent inpainting results.

Abstract

Stereo video inpainting, which aims to fill the occluded regions of warped videos with visually coherent content while maintaining temporal consistency, remains a challenging open problem. The regions to be filled are scattered along object boundaries and occupy only a small fraction of each frame, leading to two key challenges. First, existing approaches perform poorly on such tasks due to the scarcity of high-quality stereo inpainting datasets, which limits their ability to learn effective inpainting priors. Second, these methods apply equal processing to all regions of the frame, even though most pixels require no modification, resulting in substantial redundant computation. To address these issues, we introduce three interconnected components. We first propose Gradient-Aware Parallax Warping (GAPW), which leverages backward warping and the gradient of the coordinate mapping function to obtain continuous edges and smooth occlusion regions. Then, a Parallax-Based Dual Projection (PBDP) strategy is introduced, which incorporates GAPW to produce geometrically consistent stereo inpainting pairs and accurate occlusion masks without requiring stereo videos. Finally, we present Sparsity-Aware Stereo Inpainting (SASI), which reduces over 70% of redundant tokens, achieving a 10.7x speedup during diffusion inference and delivering results comparable to its full-computation counterpart, enabling real-time processing of HD (768 x 1280) videos at 25 FPS on a single A100 GPU.