One-Click Upgrade from 2D to 3D: Sandwiched RGB-D Video Compression for Stereoscopic Teleconferencing

TL;DR

This paper introduces a neural network-based method to upgrade 2D video codecs for stereo RGB-D video compression, achieving about 30% bit-rate savings while maintaining high rendering quality, without hardware changes.

Contribution

It presents a novel neural pre- and post-processing framework that enables existing 2D codecs to efficiently compress stereo RGB-D videos, with a geometry-aware loss function for better rendering.

Findings

Achieves 30% bit-rate reduction compared to MV-HEVC.

Works with various video codecs out-of-the-box.

Generalizes well to synthetic and real stereo RGB-D videos.

Abstract

Stereoscopic video conferencing is still challenging due to the need to compress stereo RGB-D video in real-time. Though hardware implementations of standard video codecs such as H.264 / AVC and HEVC are widely available, they are not designed for stereoscopic videos and suffer from reduced quality and performance. Specific multiview or 3D extensions of these codecs are complex and lack efficient implementations. In this paper, we propose a new approach to upgrade a 2D video codec to support stereo RGB-D video compression, by wrapping it with a neural pre- and post-processor pair. The neural networks are end-to-end trained with an image codec proxy, and shown to work with a more sophisticated video codec. We also propose a geometry-aware loss function to improve rendering quality. We train the neural pre- and post-processors on a synthetic 4D people dataset, and evaluate it on both synthetic and real-captured stereo RGB-D videos. Experimental results show that the neural networks generalize well to unseen data and work out-of-box with various video codecs. Our approach saves about 30% bit-rate compared to a conventional video coding scheme and MV-HEVC at the same level of rendering quality from a novel view, without the need of a task-specific hardware upgrade.