Immersive Video Compression using Implicit Neural Representations

TL;DR

This paper introduces MV-HiNeRV, an innovative INR-based codec for immersive multi-view videos, which effectively exploits redundancies to achieve significant compression gains over existing methods.

Contribution

It extends INR-based video compression to immersive videos by developing MV-HiNeRV, incorporating view-specific feature grids and shared parameters for improved redundancy exploitation.

Findings

Achieves up to 72.33% coding gains over TMIV.

Effectively exploits spatio-temporal and inter-view redundancies.

Outperforms existing immersive video codecs in tests.

Abstract

Recent work on implicit neural representations (INRs) has evidenced their potential for efficiently representing and encoding conventional video content. In this paper we, for the first time, extend their application to immersive (multi-view) videos, by proposing MV-HiNeRV, a new INR-based immersive video codec. MV-HiNeRV is an enhanced version of a state-of-the-art INR-based video codec, HiNeRV, which was developed for single-view video compression. We have modified the model to learn a different group of feature grids for each view, and share the learnt network parameters among all views. This enables the model to effectively exploit the spatio-temporal and the inter-view redundancy that exists within multi-view videos. The proposed codec was used to compress multi-view texture and depth video sequences in the MPEG Immersive Video (MIV) Common Test Conditions, and tested against the MIV Test model (TMIV) that uses the VVenC video codec. The results demonstrate the superior performance of MV-HiNeRV, with significant coding gains (up to 72.33\%) over TMIV. The implementation of MV-HiNeRV is published for further development and evaluation.