A CNN-based Post-Processor for Perceptually-Optimized Immersive Media Compression

TL;DR

This paper introduces a CNN-based post-processing technique for immersive media compression that improves quality and efficiency by resampling and reconstructing multi-view videos, demonstrating significant performance gains with VVC codec.

Contribution

The paper presents a novel CNN-based post-processor that enhances immersive media compression by optimizing spatial resolution resampling and artifact reduction.

Findings

Average BD-VMAF improvement of 3.07 across sequences

Significant rate-quality performance gains with the proposed method

Effective integration with TMIV and VVC codec

Abstract

In recent years, resolution adaptation based on deep neural networks has enabled significant performance gains for conventional (2D) video codecs. This paper investigates the effectiveness of spatial resolution resampling in the context of immersive content. The proposed approach reduces the spatial resolution of input multi-view videos before encoding, and reconstructs their original resolution after decoding. During the up-sampling process, an advanced CNN model is used to reduce potential re-sampling, compression, and synthesis artifacts. This work has been fully tested with the TMIV coding standard using a Versatile Video Coding (VVC) codec. The results demonstrate that the proposed method achieves a significant rate-quality performance improvement for the majority of the test sequences, with an average BD-VMAF improvement of 3.07 overall sequences.