Elastic3D: Controllable Stereo Video Conversion with Guided Latent Decoding

TL;DR

Elastic3D is a novel end-to-end method that converts monocular videos into high-quality, controllable stereo videos using guided latent diffusion, avoiding artifacts and providing user-adjustable stereo strength.

Contribution

It introduces a guided VAE decoder within a latent diffusion framework for artifact-free, controllable stereo video conversion from monocular videos.

Findings

Outperforms traditional warping-based methods.

Sets new standards in controllable stereo video quality.

Demonstrates effectiveness on real-world datasets.

Abstract

The growing demand for immersive 3D content calls for automated monocular-to-stereo video conversion. We present Elastic3D, a controllable, direct end-to-end method for upgrading a conventional video to a binocular one. Our approach, based on (conditional) latent diffusion, avoids artifacts due to explicit depth estimation and warping. The key to its high-quality stereo video output is a novel, guided VAE decoder that ensures sharp and epipolar-consistent stereo video output. Moreover, our method gives the user control over the strength of the stereo effect (more precisely, the disparity range) at inference time, via an intuitive, scalar tuning knob. Experiments on three different datasets of real-world stereo videos show that our method outperforms both traditional warping-based and recent warping-free baselines and sets a new standard for reliable, controllable stereo video conversion. Please check the project page for the video samples https://elastic3d.github.io.