Implicit-explicit Integrated Representations for Multi-view Video Compression

TL;DR

This paper introduces a novel multi-view video compression method combining explicit 2D video codecs with implicit neural representations, achieving high-quality reconstructions and improved compression efficiency.

Contribution

It proposes an integrated implicit-explicit framework that leverages both explicit 2D codecs and implicit neural representations for enhanced multi-view video compression.

Findings

Achieves comparable or superior compression performance to state-of-the-art standards.

Effectively reconstructs multi-view videos with high quality using the combined approach.

Demonstrates improved scene modeling and view compression on public datasets.

Abstract

With the increasing consumption of 3D displays and virtual reality, multi-view video has become a promising format. However, its high resolution and multi-camera shooting result in a substantial increase in data volume, making storage and transmission a challenging task. To tackle these difficulties, we propose an implicit-explicit integrated representation for multi-view video compression. Specifically, we first use the explicit representation-based 2D video codec to encode one of the source views. Subsequently, we propose employing the implicit neural representation (INR)-based codec to encode the remaining views. The implicit codec takes the time and view index of multi-view video as coordinate inputs and generates the corresponding implicit reconstruction frames.To enhance the compressibility, we introduce a multi-level feature grid embedding and a fully convolutional architecture into the implicit codec. These components facilitate coordinate-feature and feature-RGB mapping, respectively. To further enhance the reconstruction quality from the INR codec, we leverage the high-quality reconstructed frames from the explicit codec to achieve inter-view compensation. Finally, the compensated results are fused with the implicit reconstructions from the INR to obtain the final reconstructed frames. Our proposed framework combines the strengths of both implicit neural representation and explicit 2D codec. Extensive experiments conducted on public datasets demonstrate that the proposed framework can achieve comparable or even superior performance to the latest multi-view video compression standard MIV and other INR-based schemes in terms of view compression and scene modeling.