Lightweight Multiplane Images Network for Real-Time Stereoscopic Conversion from Planar Video

TL;DR

This paper introduces a lightweight, real-time stereoscopic conversion network using multi-plane images that balances high-quality reconstruction with efficient inference, suitable for 2K resolution virtual reality and 3D displays.

Contribution

It proposes a novel multi-plane image network with a lightweight depth-semantic branch and a simplified MPI rendering process for real-time stereoscopic conversion.

Findings

Achieves real-time 2K resolution inference.

Provides over 40x acceleration compared to SOTA TMPI.

Maintains comparable visual quality to state-of-the-art models.

Abstract

With the rapid development of stereoscopic display technologies, especially glasses-free 3D screens, and virtual reality devices, stereoscopic conversion has become an important task to address the lack of high-quality stereoscopic image and video resources. Current stereoscopic conversion algorithms typically struggle to balance reconstruction performance and inference efficiency. This paper proposes a planar video real-time stereoscopic conversion network based on multi-plane images (MPI), which consists of a detail branch for generating MPI and a depth-semantic branch for perceiving depth information. Unlike models that depend on explicit depth map inputs, the proposed method employs a lightweight depth-semantic branch to extract depth-aware features implicitly. To optimize the lightweight branch, a heavy training but light inference strategy is adopted, which involves designing a coarse-to-fine auxiliary branch that is only used during the training stage. In addition, the proposed method simplifies the MPI rendering process for stereoscopic conversion scenarios to further accelerate the inference. Experimental results demonstrate that the proposed method can achieve comparable performance to some state-of-the-art (SOTA) models and support real-time inference at 2K resolution. Compared to the SOTA TMPI algorithm, the proposed method obtains similar subjective quality while achieving over $40\times$ inference acceleration.